Image processing device for generating reconstruction image, image generating method, and storage medium

ABSTRACT

A sub-image extractor extracts a target sub-image from a light field image. A partial area definer defines a predetermined area in the target sub-image as a partial area. A pixel extractor extracts pixels from the partial area, the number of pixels meeting correspondence areas of a generation image. The pixel arranger arranges the extracted pixels to the correspondence areas of the generation image in an arrangement according to the optical path of the optical system which photographs the light field image. Pixels are extracted for all sub-images in the light field image, and are arranged to the generation image to generate a reconstruction image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefits of Japanese Patent Application No.2011-285188, filed on Dec. 27, 2011, Japanese Patent Application No.2011-284969, filed on Dec. 27, 2011, Japanese Patent Application No.2011-287041, filed on Dec. 27, 2011, and Japanese Patent Application No.2012-169655, filed on Jul. 31, 2012, the entire disclosure of which isincorporated by references herein.

FIELD

The present invention relates to a technique which reconstructs animage.

BACKGROUND

In recent years, an imaging device which captures information ondirection distribution of incident light, that is an imaging devicecalled as “plenoptic camera”, is known.

In optical systems of the plenoptic camera, a compound eye-like lens(hereinafter, referred to as “micro lens array”), in which extremelysmall lenses (hereinafter, referred to as “micro lenses”) are repeatedlyarranged, is inserted between a conventional imaging lens (hereinafter,referred to as “main lens”) and an imaging element.

Each micro lens which forms a micro lens array distributes the lightfocused by the main lens to a plurality of pixel groups in an imagingelement according to the angle of the received light.

That is, if an image focused on an imaging element by each micro lens ishereinafter referred to as “sub-image,” data of image formed by anaggregate of a plurality of sub-images is output from an imaging elementas data of a captured image.

Such captured image by the plenoptic camera is hereinafter referred toas “light field image.”

The light field image is generated by the light which is incidentthrough not only a conventional main lens but also the micro lens array.Therefore, the light field image includes two-dimensional spacialinformation which indicates from which part the light beam reaches andis included in conventional captured images, and further includestwo-dimensional direction information indicating a direction from whichthe light beam reaches when viewed from imaging element, as informationnot included in conventional captured images.

The plenoptic camera, after imaging the light field image, canreconstruct an image on a plane separated at an arbitrary distance aheadat the time of imaging using the data of the light field image.

In other words, the plenoptic camera can freely make data of an image(hereinafter, referred to as “reconstruction image”) by using data oflight field image after the imaging as if the image is focused atpredetermined distance and captured, even if the light field image iscaptured without focusing at predetermined distance.

SUMMARY

An image processing device according to a first aspect of the presentinvention comprises: an image obtainer that obtains a multi-view imagein which sub-images from viewpoints are aligned; a clipper that clipspartial images from the sub-images; and a generator that generates areconstruction image by arranging the partial images based on analignment position of the sub-images corresponding to the partialimages.

An image processing device according to a second aspect of the presentinvention comprises: an image obtainer that obtains a multi-view imagein which sub-images from viewpoints are aligned; a first selector thatselects a first image from the sub-images; a pixel selector that selectsa first pixel included in the first image; a second selector thatselects a second image from the sub-images; a searcher that searches asecond pixel corresponding to the first pixel among pixels included inthe second image; a pixel processor that processes the first pixel andthe second pixel based on a position relationship between the firstpixel and the second pixel; and a reconstructor that reconstructs animage from the sub-images processed by the pixel processor.

An image generating method according to a third aspect of the presentinvention comprises steps of: obtaining a multi-view image in whichsub-images from viewpoints are aligned; clipping partial images from thesub-images; and arranging the partial images based on an alignmentposition of the sub-images corresponding to the partial images togenerate the reconstruction image.

A non-transitory computer-readable storage medium according to a fourthaspect of the present invention stores a computer-executable program,the program causes a computer to execute: obtaining a multi-view imagein which sub-images from viewpoints are aligned; clipping partial imagesfrom the sub-images; and arranging the partial images based on analignment position of the sub-images corresponding to the partial imagesto generate the reconstruction image.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of this application can be obtained whenthe following detailed description is considered in conjunction with thefollowing drawings, in which:

FIG. 1A is an front view of a digital camera according to an embodiment1 of the present invention;

FIG. 1B is a back view of the digital camera according to the embodiment1;

FIG. 2 is a block diagram illustrating a hardware configuration of thedigital camera according to the embodiment 1;

FIG. 3 is an exemplary diagram illustrating a configuration example ofan optical system in the digital camera according to the embodiment 1;

FIG. 4 is a conceptual diagram of a light field image according to theembodiment 1;

FIG. 5 is a block diagram illustrating a functional constitution of thedigital camera according to the embodiment 1;

FIG. 6 is a flowchart of an image output process according to theembodiment 1;

FIG. 7 is a block diagram illustrating a functional constitution of alive view image generator according to the embodiment 1;

FIG. 8 is a flowchart of an image generation process executed by thedigital camera according to the embodiment 1;

FIG. 9 is a drawing illustrating a concept of a process which generatesa first reconstruction image from the light field image according to theembodiment 1;

FIG. 10A is a block diagram illustrating a functional constitution of aconfirmation image generator according to the embodiment 1;

FIG. 10B is a drawing illustrating an example of an area definitiontable according to the embodiment 1;

FIG. 11 is a flowchart of an image generation process executed by thedigital camera according to the embodiment 1;

FIG. 12 is a drawing illustrating a concept of a process which generatesa second reconstruction image from the light field image according tothe embodiment 1;

FIG. 13A is a block diagram illustrating a functional constitution of amain image generator according to the embodiment 1;

FIG. 13B is a block diagram illustrating a functional constitution of anpixel displacement degree calculator according to the embodiment 1;

FIG. 13C is a drawing illustrating an example of an arrangement intervaltable according to the embodiment 1;

FIG. 13D is a drawing illustrating an example of a clipping sizecorrection table according to the embodiment 1;

FIG. 14 is a flowchart of an image generation process executed by thedigital camera according to the embodiment 1;

FIG. 15 is a drawing illustrating a concept of a process which generatesa third reconstruction image from the light field image according to theembodiment 1;

FIG. 16A and FIG. 16B are drawings illustrating a concept of a processwhich arranges partial images in a process which generates a thirdreconstruction image according to the embodiment 1;

FIG. 17 is a flowchart of an image output process according to anembodiment 2 of the present invention;

FIG. 18 is a block diagram illustrating a functional constitution of alive view image generator according to an embodiment 3 of the presentinvention;

FIG. 19 is a flowchart of an image output process according to theembodiment 3;

FIG. 20 is a flowchart of an image generation process executed by thedigital camera according to the embodiment 3;

FIG. 21 is a block diagram illustrating a functional constitution of alive view image generator according to an embodiment 4 of the presentinvention;

FIG. 22 is a flowchart of an image output process according to theembodiment 4;

FIG. 23 is a flowchart of a displacement degree estimation processaccording to the embodiment 4;

FIG. 24 is a drawing illustrating an example of grouping sub-images toestimation group and calculation group according to the embodiment 4;

FIG. 25 is a drawing illustrating an example of grouping sub-images toestimation images and calculation images according to the embodiment 4;

FIG. 26 is a drawing illustrating an example of an estimation process ofa displacement degree according to the embodiment 4;

FIG. 27A is a block diagram illustrating a functional constitution of amain image generator according to an embodiment 5 of the presentinvention;

FIG. 27B is a drawing illustrating an example of a correspondence tableof a re-focal length, a reconstruction distance, and a clipping sizeaccording to the embodiment 5;

FIG. 28 is a flowchart of an image generation process according to theembodiment 5;

FIG. 29 to FIG. 31 are drawings illustrating a concept of a processwhich arranges partial images on an intermediate image according to theembodiment 5;

FIG. 32 A to FIG. 32C are drawings illustrating examples of arrangementsof the partial images on the intermediate image according to theembodiment 5;

FIG. 33 is a flowchart of an image generation process 7 according to amodification of the embodiment 5;

FIG. 34 is a drawing illustrating a concept of a process which arrangesthe partial images on the intermediate image according to themodification of the embodiment 5; and

FIG. 35 is a drawing illustrating a concept of a process which arrangesthe partial images on the intermediate image according to themodification of the embodiment 5.

DETAILED DESCRIPTION

Hereinafter, an imaging device according to embodiments of the presentinvention will be described with reference to drawings. Besides, theidentical reference is given to the identical or corresponding part inthe drawings.

(Embodiment 1)

An external appearance of a digital camera 1 as one embodiment of theimaging device according to the present invention will be described.FIG. 1A illustrates an external appearance of a front side. FIG. 1Billustrates an external appearance of a back side. The digital camera 1includes imaging lenses (lens group) 12 in the front side. The back sideof the digital camera 1 is provided with a liquid crystal displaymonitor 13 as a display, and an operator 14 including a mode dial, acursor key, a SET key and zoom buttons (W button and T button), a menukey, and the like. Moreover, a shutter key 10 and a power button 11 areprovided on a top surface. Besides, although not illustrated, the sidepart is provided with a USB terminal connector used when connecting withan external device such as a personal computer (hereinafter, PC) and amodem via USB cables.

Next, a hardware configuration of the digital camera 1 will beillustrated with reference to FIG. 2.

The digital camera 1 includes a CPU (Central Processing Unit) 21, a ROM(Read Only Memory) 22 and a RAM (Random Access Memory) 23, and aninternal bus 20. The digital camera 1 includes an I/O interface 30, animager 31, an input device 32, an output device 33, a memory 34, adisplay 35, a communicator 36, and a media drive 37.

The CPU 21 performs a process for outputting image, which is describedbelow, in accordance with a program stored in the ROM 22 or the memory34, using the RAM 23 as workspace.

At least one of the ROM 22 and memory 34 stores data required forperforming various kinds of processes by the CPU 21, the processes beingdescribed below. The stored data is arbitrarily loaded to the RAM 23.Moreover, the RAM 23 timely stores the intermediate data of the processmentioned below.

The CPU 21, the ROM 22, the RAM 23, and the I/O interface 30 areconnected one another through the internal bus 20. Moreover, the imager31, the input device 32, the output device 33, the memory 34, thedisplay 35, the communicator 36, and the media drive 37 are connected tothe I/O interface 30.

The imager 31 includes a main lens 311, a micro lens array 312, and animaging element 313. Further details of the imager 31 are describedbelow with reference to FIG. 3.

The input device 32 includes input devices and a transmitter, the inputdevices being various buttons of a shutter key 10 and an operator 14,and a touch panel provided on the display 35 and the like, and thetransmitter transferring information on the operation performed by theuser using the input devices to the I/O interface 30. The user can entera command to the digital camera 1 using the input device 32, and canenter various kinds of information.

The output device 33 includes a monitor, loudspeaker and the like, andoutputs various images and various voices which are generated by theprocess of the CPU 21.

The memory 34 includes a hard disk, a DRAM (Dynamic Random AccessMemory) and the like, and stores the data of various images and variouskinds of setting information, such as a light field image and areconstruction image which are mentioned below, transferred from the CPU21 or input from other equipment.

The communicator 36 controls a communication with other equipment (notillustrated) through a network including the Internet.

A removable media 38 (a magnetic disc, an optical disc, amagneto-optical disc, semiconductor memory, or the like) is insertedarbitrarily in the media drive 37. A program retrieved from theremovable media 38 by the media drive 37 is installed in the memory 34if necessary. Moreover, the removable media 38 can store the variouskinds of data such as the data of the images stored in the memory 34, asis the case with the memory 34. The display 35 includes a liquid crystaldisplay, an electro-luminescence display, or the like. The display 35displays the live view images prior to photographing (in a photographingpreparatory phase or live view mode) and an image for confirmation afterphotographing, which are generated by the process described later andtransferred from the CPU 21. Hereinafter, it is described assuming thatthe display 35 has the resolution of 320×240 pixels.

With respect to the digital camera 1 with such configuration, aconfiguration example of an optical system therein will be describedwith reference to FIG. 3.

In the optical system of the digital camera 1, a main lens 311, a microlens array 312, and imaging element 313 are arranged in this order whenviewed from photographic object OB.

In the micro lens array 312, N×M pieces of micro lenses 312-1 to 312-N×M(N and M are two or more arbitrary integer) are arranged. FIG. 3illustrates micro lenses (N pieces) in one column of horizontaldirection. Hereinafter, it is described assuming that the micro lenses312-i (i is an integer within a range of 1 to N×M) have the samediameter, and are arranged in a lattice pattern at the same interval inthe micro lens array 312.

The main lens 311 condenses a light flux emitted from a point on thephotographic object OB to form an image on a predetermined plane MA, andcauses the light flux to be entered in the micro lens array 312.Hereinafter, the plane on which the image is formed by the main lens 311is referred to as “main lens imaging plane MA.” In the presentembodiment, the main lens imaging plane MA is presented between the mainlens 311 and the imaging element 313.

The micro lenses 312-i condenses the light flux which is entered throughthe main lens 311 from the photographic object OB for each incidencedirection, to form sub-images on the imaging element 313.

In other words, in the imaging element 313, a plurality of sub-imagesare formed by each of a plurality of micro lenses 312-1 to 312-N×M, andthe light field image is generated which is an aggregate of theplurality of sub-images.

Thus, the images seen from different angles to the identicalphotographic object are recorded as a plurality of correspondingsub-images. In other words, an image is obtained in which a plurality ofsub-images when the photographic object is seen from a plurality ofviewpoints at the same time are aligned.

The sub-images obtained in this manner includes images when theidentical photographic object is seen from different angles, differentviewpoints. Therefore, selecting and combining suitable pixels from aplurality of sub-images allows a reconstruction of an image imaged byfocusing on a plane separated at arbitrary distance ahead when imaging,and an image the depth of field of which is arbitrarily changed.

An example of the light field image LFI which is obtained byphotographing the photographic object OB of a block shape is illustratedin FIG. 4.

The light field image LFI includes the images (sub-images S₁₁ to S_(MN))which correspond to N×M pieces of micro lenses 312-i arranged in thelattice pattern, respectively. For example, the upper left sub-image S₁₁corresponds to the image when the photographic object OB is photographedfrom the upper left, and the lower right sub-image S_(MN) corresponds tothe image when the photographic object OB is photographed from the lowerright. Images in which a plurality of sub-images from a plurality ofviewpoints is aligned like the light field image LFI are multi-viewimages.

The sub-images on the i-th row (sub-images in one row in horizontaldirection) S_(i1)-S_(iN) correspond to stereo images in which the imageformed by the main lens 311 is imaged by the micro lenses 312-i arrangedin i-th row in the micro lens array 312. Similarly, the sub-images onthe j-th column (sub-images in one column in vertical direction)S_(1j)-S_(Mj) correspond to stereo images in which the image formed bythe main lens 311 is imaged by the micro lenses 312-i arranged in j-thcolumn in the micro lens array 312.

In the present embodiment, each sub-image S is a gray scale image, andeach pixel which forms the sub-image has a pixel value (scalar value).

In the present embodiment, 80 pieces of micro lenses are arranged inhorizontal direction (x-axis), and 60 pieces of micro lenses arearranged in vertical direction (y-axis), in the micro lens array 312.Each sub-image corresponding to each micro lens is in an area with 60×60pixels to form the light field image LFI which has the resolution of4800×3600 pixels.

In the optical system as illustrated in FIG. 3, the focal point of themain lens is presented in the main lens side from the imaging elementand the focal point of the micro lenses is presented behind the imagingelement. In the light field image LFI photographed by such opticalsystem, the photographic object is captured as reverted image which isinverted point-symmetric manner in the unit of sub-image. On the otherhand, by an optical system which has a focal point of the main lensbehind the imaging element, the photographic object is captured as animage which is not inverted on the sub-image. This is the same when thefocal point of the main lens is in the main lens side from the imagingelement, and the focal point of the micro lenses is in front of theimaging element. The digital camera 1 generates the reconstruction imageRI from such light field image LFI and outputs the reconstruction image.

The digital camera 1 functions as, according to the above-describedphysical configuration, a photographing device 40, an input device 50,an image generation processor 60 including an LFI generator 610, a liveview image generator 620, a confirmation image generator 630 and a mainimage generator 640, a display 70, and a memory 80, as illustrated inFIG. 5.

The photographing device 40 photographs the photographic object by themain lens 311 and the micro lens array 312, and transfers theinformation to the LFI generator 610. The input device 50 acquiresphotographing setting stored in the ROM 22 and the RAM 23, a user'soperation received by the input device 32, and reconstruction settingstored in the ROM 22 or the RAM 23, and transfers acquired informationto each element of the image generation processor 60.

The LFI generator 610 generates the light field image LFI (multi-viewimage), which is exemplified in FIG. 4, from the information transferredfrom the photographing device 40. The LFI generator 610 transfers thegenerated light field image LFI to the live view image generator 620,the confirmation image generator 630, and the main image generator 640.

The live view image generator 620, the confirmation image generator 630,and the main image generator 640 generate the reconstruction image fromthe sub-images of the light field image LFI based on the position ofeach micro lens.

The process to generate the reconstruction image based on the positionof each micro lens from the light field image is hereinafter referred toas “reconstruction process.” In the reconstruction process, a surface onwhich the photographic object to be reconstructed is virtually presentedis referred to as “reconstruction surface.”

The live view image generator 620 generates images for live view (liveview images) among the reconstruction images. The process to generatethe live view images is referred to as “live view image generationprocess.” In the live view mode prior to photographing, it is desired todisplay the images of the current photographic object switching rapidlyfor photographing preparation. For this reason, the live view imagegeneration process is made to use small calculations in the presentembodiment. The live view image generator 620 outputs the generated liveview image to the display 70. Since the live view image generator 620generates the images used for the live view mode prior to photographing(preparation of photographing), the live view image generator 620 isalso referred to as a preparation generator or a first generator.

The confirmation image generator 630 generates the image for theconfirmation (confirmation image) after image photographing among thereconstruction images. The process to generate the confirmation image isreferred to as a “confirmation image generation process.” In theconfirmation after photographing, image quality like the image forviewing is not required, whereas suitable degree of image quality isdesired so that it is possible to confirm what the obtained image islike. Moreover, although it may be necessary to display the confirmationimage promptly after photographing, there is no necessity to generate animage as quickly as in the live view mode. For this reason, aconfirmation image generation process is made to use more calculationsthan live view image generation process, so that the confirmation imagegeneration process can generate the image with higher image quality. Theconfirmation image generator 630 outputs the generated confirmationimage to the display 70. High image quality herein not only representshigh resolution, but also generally indicates images which areconvenient for a user, for example, image with high accuracy ofreconstruction, image with low noises, and image in which suitableblurring is added for emphasize the non-blurred features and emphasizethe blurred features of the photographic objects. The confirmation imagegenerator 630 generates the image for a display in order to confirm thephotographed image, and the confirmation image generator 630 istherefore referred to as display generator or a second generator aswell.

The main image generator 640 generates the image for viewing (mainimage) among the reconstruction images. The process to generate the mainimage is referred to as “main image generation process.” In order tosatisfy the needs for viewing, the main image is desired that the imagemeets the reconstruction setting requested by a user, and has a highimage quality. For this reason, the main image generation is made to usemany calculations. The main image generator 640 stores the generatedmain image to the memory 80. The main image generator 640 is referred toas a third generator as well. It is noted that the live view imagegenerator 620, the confirmation image generator 630, and the main imagegenerator 640 may not be distinguished, and may be simply referred to asa generator. Moreover, when the reconstruction image generated by theconfirmation image generator 630 (the second generator) satisfies a needof user's image quality, the confirmation image generator 630 may alsoserve as the main image generator 640 (a third generator).

The display 70 displays the transferred reconstruction images. Thedisplay 70 has the resolution of 320×240 pixels in the embodiment.

The memory 80 stores the transferred main image. The memory 80 may storethe main image with a corresponding confirmation image.

The process (image output process 1) for acquiring the light field imageLFI and outputting the reconstruction image, which is performed by thedigital camera 1, will be described with reference to FIG. 6.

When the digital camera 1 is powered on and the input device 50 receivesan operation to prepare the photographing, the digital camera 1 startsthe image output process 1 illustrated in FIG. 6.

In the image output process 1, the digital camera 1 first acquires thelight field image LFI using the photographing device 40 and LFIgenerator 610 (step S101).

Next, the input device 50 acquires current imaging setting informationstored in the RAM 23 and the ROM 22 (step S102). The imaging setupinformation includes current focal length f_(ML) of the main lens 311,diaphragm (F value), position information of each micro lens in themicro lens array 312, position information of the micro lens array 312and the imaging element 313, and the like.

When the imaging setting information is acquired, the live view imagegenerator 620 performs a live view image generation process (step S103).In the present embodiment, the live view image generation process is animage generation process 1 described below.

The functional constitution of the live view image generator 620 whichperforms the live view image generation process will be described withreference to FIG. 7. In the present embodiment, the live view imagegenerator 620 corresponds to a first reconstruction image generator 710illustrated in FIG. 7.

The first reconstruction image generator 710 includes a sub-imageextractor 711, a partial area definer 712, a pixel extractor 713, apixel arranger 714, and an output device 715. By means of such elements,the first reconstruction image generator 710 generates a firstreconstruction image (live view image of the present application) as anoutput image from the light field image LFI. Hereinafter, it isdescribed with an example of a case where the first reconstruction imageis an image of 320×240 pixels.

The sub-image extractor 711 extracts one sub-image from the sub-imageswhich forms the light field image LFI as a target sub-image in order,and transfers the target sub-image to the partial area definer 712.

The partial area definer 712 defines a predetermined area in the targetsub-image transferred from the sub-image extractor 711 as a partialarea. The details of the partial area are described below. The partialarea definer 712 transfers the information indicating the partial areato the pixel extractor 713.

The pixel extractor 713 extracts pixels, in accordance with apredetermined condition, from the partial area on the target sub-imagewhich is indicated by the information transferred from the partial areadefiner 712. The concrete steps of the process which extracts the pixelsare described below. The pixel extractor 713 transfers the extractedpixels to the pixel arranger 714. As identifying the portion (partialarea) of the sub-image which is either inside or outside a region(clipping partial area from sub-image) and extracting pixel data on thepartial area, the partial area definer 712 and pixel extractor 713 alsobehave as a “clipper” as a whole.

The pixel arranger 714 arranges the pixels extracted by the pixelextractor 713 on the first reconstruction image. The concrete steps ofthe process which arranges the pixels are described below. If an imageson a partial area is referred to as a “partial image,” and an arrangingpixel of a partial image is referred to as an “arranging a partialimage,” the pixel arranger 713 can be referred to as an “arrangerarranging the partial image on the first reconstruction image”.

The sub-image extractor 711 through the pixel arranger 71 generates thefirst reconstruction image, by extracting pixels using all sub-images asthe target images in order, and arranging the pixels on the firstreconstruction image.

The output device 715 outputs the first reconstruction image generatedin this way to outside (the display 70).

Next, a reconstruction image generation process (image generationprocess 1) performed by the first reconstruction image generator 710will be described with reference to FIG. 8 and FIG. 9.

The live view image generator 620 (first reconstruction image generator710), when arriving at step S103 of the image output process 1 (FIG. 6),starts the image generation process 1 illustrated in FIG. 8.

In the image generation process 1, k1 represents a counter variable, andthe sub-image extractor 711 first extracts k1-th sub-image in the lightfield image LFI as a target sub-image (step S201).

Next, the partial area definer 712 defines the predetermined area (here,area with a square of 8×8 pixels located at the center of the sub-image)in the target sub-image as the partial area (step S202). The size ofpredetermined area is determined based on the range of displacement(parallax) of the pixels corresponding to photographic objects on therange of object distance designed to the digital camera 1. For example,when the range of displacement of the pixels corresponding to thenearest and farthermost objects on the range of object distance designedto the camera is from 4 pixels to 12 pixels, approximately intermediatepixels such as 8 pixels are adopted.

When the partial area is defined, the pixel extractor 713 extracts thepixels of the partial area (step S203). The number of extracted pixelsis the number of pixels in an area (correspondence area) correspondingto the target sub-image in the first reconstruction image which isdescribed below.

Then, the pixel arranger 714 arranges the extracted pixels tocorrespondence parts of the corresponding area on the firstreconstruction image (generation image) (step S204).

The process from step S201 to step S204 will be described with referenceto FIG. 9 using concrete examples.

As illustrated in the upper side of FIG. 9, in the light field imageLFI, the approximately circular sub-images S₁₁ to S_(MN) are arrangedwith the lattice pattern. The sub-images are extracted as the targetsub-images from the sub-image S₁₁ in order (step S201), and the partwith a square of 8×8 pixels at the central part is extracted as thepartial area (step S202).

The enlargements of the partial area (partial image PI₁₁) extracted fromthe sub-image S₁₁ and the partial area (partial image PI₂₃) extractedfrom the sub-image S₂₃ are illustrated in the middle of FIG. 9. Thepixels of each partial area are represented by a lattice, and the pixelsis represented by the coordinates (1, 1) to (8, 8).

The example of the first reconstruction image RI is illustrated in thelower side of FIG. 9. In this example, the first reconstruction image isan image with 320×240 pixels, and each of 80×60 pieces of sub-images isdivided into a correspondence area with 4×4 pixels (area illustrated bya thick line) according to the coordinates of the sub-image.

The corresponding area of S₁₁ is a square part with coordinates (1, 1)to (4, 4) in the first reconstruction image RI, and the correspondencearea of S₂₃ is a square part with coordinates (9, 5) to (12, 8).

At step S203, 4×4 pixels are extracted every other pixels from thepartial area. Then, at step S204, the position of the extracted pixelsis inverted in four directions (point symmetry) for every sub-image, andthe pixels are arranged at the inverted position on the correspondencearea. This is because the image of the photographic object OB has beenreversed for every sub-image, since the main lens imaging plane MA ispresented in front of the imaging element 313.

In the example of FIG. 9, the pixel (1, 1) in the partial area of S₁₁ isarranged at the coordinate (4, 4) in the first reconstruction image RI.Moreover, the pixel (1, 3) in the partial area of S₁₁ is arranged at thecoordinate (3, 4) in the first reconstruction image RI. In a similarway, every other pixels are extracted from the partial area andarranged, and the pixel (7, 7) in the partial area is arranged at thecoordinate (1, 1) in the first reconstruction image RI.

Return to FIG. 8, when the extracted pixels are arranged to all pixelsof the correspondence area with respect to the current target sub-imageat step S204, it is determined whether the above-described processeshave been performed on all sub-images (step S205). When an unprocessedsub-image exists (step S205; NO), the counter variable k1 is incremented(step S206), and the process is repeated from step S201 for thefollowing target sub-image.

On the other hand, when the above-described process have been performedfor all sub-images (step S205; YES), the completed first reconstructionimage is regarded as the live view image, and then the image generationprocess 1 is terminated.

In the case where the number of pixels in the partial area is the sameas the number of pixels in the correspondence area, the above-describedextraction process may be omitted, and all pixels in the partial areaare inverted in point symmetry manner to arrange the pixels at theinverted position on the correspondence area.

Moreover, when the pixels cannot be extracted at equal interval, forexample in the case where the partial area has 11×11 pixels and thecorrespondence area has 4×4 pixels, the pixels are extracted in anextraction method of approximately equal intervals, such that the firstpixel, the fourth pixel, the seventh pixel, and the eleventh pixel areextracted. Alternatively, the partial area (11×11 pixels) may be resizedto the correspondence area (4×4 pixels) with interpolation.

Return to FIG. 6, after generating the live view image, the display 70displays the generated live view image (step S104).

Then, it is determined whether or not the input device 50 detects animaging (photographing) operation (operation of pressing the shutter key10) (step S105). When it is determined that the imaging operation is notdetected (step S105; NO), the digital camera 1 returns to step S101 andrepeats the process for displaying the live view image.

On the other hand, when it is determined that the imaging(photographing) operation is detected (step S105; YES), the confirmationimage generator 630 next performs the confirmation image generationprocess (step S106). In the present embodiment, the confirmation imagegeneration process corresponds to the image generation process 2described below.

The functional constitution of the confirmation image generator 630which performs the confirmation image generation process will describedwith reference to FIG. 10A. In the present embodiment, the confirmationimage generator 630 corresponds to a second reconstruction imagegenerator 720 illustrated in FIG. 10A.

The second reconstruction image generator 720 includes a sub-imageextractor 721, a displacement value calculator 722, a partial areadefiner 723, an arrangement image generator 724, an image arranger 725,and an output device 726. By means of such elements, the secondreconstruction image generator 720 generates the second reconstructionimage (the confirmation image in the present embodiment) as an outputimage from the light field image LFI. The second reconstruction imageis, for example, an image with 320×240 pixels.

The sub-image extractor 721 extracts one sub-image from the sub-imageswhich forms the light field image LFI as a target sub-image in order,and transfers the target sub-image to the displacement value calculator722.

The displacement value calculator 722 calculates a degree (imagedisplacement degree) indicating a degree of displacement of the targetsub-image to a sub-image in a predetermined range around the targetsub-image (peripheral sub-images).

The image displacement degree is calculated, for example by thefollowing manners.

A predetermined area near a center of the target sub-image (for example,center area of 10×10), which is determined according to a specificationof design, is referred to as a center area. A position is calculatedwhere an image corresponding to the center area of the sub-image ispresented in a sub-image on the right (if the sub-image on the rightdoes not exist, a sub-image on the left, in this case hereinafterright-and-left reversal). Although an arbitrary known manner ofdetermining a position where the part of image is presented in anotherimage may be used, the present embodiment uses the following manner.

First, center area of 10×10 in the right image is set as a calculationtarget part. Then, it is calculated the sum of absolute values of pixelvalue differences between the central part and the calculation targetpart. Next, the sum of absolute values of differences is similarlycalculated for an area shifted 1 pixel to the right. Such calculation isrepeated within a range of possible parallax. Then, a positiondisplacement of which the obtained sum of absolute values of differencesis the minimum is determined as the image displacement degree.

In case of obtaining the image displacement degree from two sub-imageson the right, the sum of absolute values of differences is calculated ina position which is shifted d pixels for the sub-image on immediateright, the sum of absolute values of differences is calculated in aposition which is shifted 2d pixels for the sub-image next to thesub-image on immediate right, the sum of two sums is set as anevaluation function, and d is obtained for which the evaluation functionis the minimum. In case of increasing the number of sub-images, it canbe similarly processed.

The partial area definer 723 acquires the partial area sizecorresponding to the image displacement degree calculated by thedisplacement value calculator 722 with reference to an area definitiontable stored in the ROM 22.

An area definition table records, as illustrated in FIG. 10B, a rangefor the image displacement degree, and the value indicating an area sizeassociated with each other. For example, in the case where the imagedisplacement degree is 8, the partial area is an area of 6×6 pixels atthe center of the target sub-image. The partial area definer 723transfers the information indicating the defined partial area to thearrangement image generator 724.

In the area definition table, it is desirable that the partial area sizeis increased generally in proportion to an increase of the imagedisplacement degree on the premise of a predetermined offset value.

A method for calculating the area size is not limited to the manners,the area size may be calculated using arbitrary tables or formulasaccording to which the partial area size (natural number) is larger inthe predetermined range when the image displacement degree is larger.

When the arrangement image generator 724 receives the information whichdefines the partial area, the arrangement image generator 724 resizesthe image included to the area with interpolation, and generates thearrangement image for arranging on the generation image (the secondreconstruction image). Specifically, the arrangement image generator 724resizes the image to a size (here 4×4 pixels) corresponding to the sizeof the correspondence area of the generation image (the secondreconstruction image). As identifying the portion (partial area) of thesub-image which is either inside or outside a region (clipping partialarea from sub-image) and extracting image on the partial area togenerate an arrangement image, the partial area definer 723 andarrangement image generator 724 also behave as a “clipper” as a whole.

The image arranger 725 arranges the arrangement image generated by thearrangement image generator 724 on the second reconstruction image.Arranging the image from the partial aria, the image arranger 725 can bereferred to as an “arranger arranging the partial image on the secondreconstruction image”.

The sub-image extractor 721 through the image arranger 725 generate thearrangement images for all sub-images, and the image arranger 725arranges the arrangement images on the second reconstruction image togenerate the second reconstruction image.

The output device 726 outputs the second reconstruction image generatedin this manner to outside (the display 70).

Next, the process (the image generation process 2) performed by thesecond reconstruction image generator 720 will be described withreference to FIG. 11 and FIG. 12.

When arriving step S106 in the image output process 1 (FIG. 6), theconfirmation image generator 630 (the second reconstruction imagegenerator 720) starts the image generation process 2 illustrated in FIG.11.

In the image generation process 2, k2 represents a counter variable, andthe sub-image extractor 721 extracts k2-th sub-image in the light fieldimage LFI as a target sub-image (step S301).

Next, the displacement value calculator 722 calculates the imagedisplacement degree in the above-described manner (step S302), the imagedisplacement degree indicating an estimation value on how many pixelsthe photographic object captured in the predetermined area (10×10 pixelsof central part) of the target sub-image are shifted from the center ofthe sub-image on the right.

Next, the partial area definer 723 defines the partial area of thetarget sub-image (step S303). At step S303, an area size L correspondingto the image displacement degree obtained at step S302 is acquired withreference to the area definition table. Then, the area having theacquired size (square area with L×L pixels) at the center of the targetsub-image is defined as the partial area.

The arrangement image generator 724 resizes the image of the partialarea having L×L pixels to fit the area (correspondence area)corresponding to the target sub-image in the second reconstruction imageto be the arrangement image (step S304). The size of the correspondencearea is a size of each sub-image which is obtained by dividing thesecond reconstruction image (here 320×240 pixels) which is determinedaccording to a setting, and follows a setting value stored in advance.

The arrangement image is arranged to the part corresponding to thetarget sub-image on the second reconstruction image (generation image)(step S305). At this time, the image is inverted in four directionsprior to the arrangement.

The concrete example of the process from step S301 to step S305 will bedescribed with reference to FIG. 12.

As illustrated in the upper side of FIG. 12, in the light field imageLFI, the approximately circular sub-images S₁₁ to S_(MN) are arrangedwith the lattice pattern. The sub-images are extracted as the targetsub-images from the sub-image S₁₁ in order (step S301), and a positiondisplacement degree with the sub-image on the right is acquired (stepS302). In FIG. 12, it is supposed that the image displacement degree ofS₁₁ to S₁₂ illustrated by a thick arrow is larger, and the imagedisplacement degree of S₂₃ to S₂₄ is small.

Next, the partial area definer 723 defines the size (clipping size) forextracting the partial area so that larger the image displacement degreeis, larger the partial area is (step S303).

As illustrated in the middle of FIG. 12 in this example, a square partwith 10×10 pixels at the central part of S₁₁ is defined as the partialarea of S₁₁, and a square part with 3×3 pixels at the central part ofS₂₃ is defined as the partial area of S₂₃, respectively.

An example of the second reconstruction image RI is illustrated in thelower side of FIG. 12. In this example, the second reconstruction imagehas 320×240 pixels, and, for each of 80×60 pieces of sub-images, thecorrespondence area with 4×4 pixels (area illustrated by a thick line)in accordance with the coordinates of the sub-image is defined.

At step S304, the image of the partial area is resized to be thearrangement image with 4×4. At following step S305, the image arranger725 inverts the arrangement image on four directions (point symmetry),and arranges the inverted image to the correspondence area. This isbecause the main lens imaging plane MA is presented in front of theimaging element 313, and thus the image of the photographic object OB isinverted for every sub-image. The resizing is only reducing size whenthe second reconstruction image corresponding to the number of pixels ofthe liquid crystal display monitor 13 is small, and a rate of reductionchanges according to the image displacement degree. On the other hand,when the number of pixels of the liquid crystal display monitor 13 islarge and the correspondence area is large, expansion or reduction isperformed depending on the image displacement degree.

Return to FIG. 11, the image has been arranged to the correspondencearea for the current target sub-image at step S305, it is thendetermined whether the above-described process is performed for allsub-images (step S306). When an unprocessed sub-image exists (step S306;NO), the counter variable k2 is incremented (step S307), and the processfrom step S301 is repeated for the following target sub-image.

On the other hand, when the above-described process has been performedfor all sub-images (step S306; YES), the completed second reconstructionimage is set as the confirmation image, and the image generation process2 is terminated.

In this example, the confirmation image has the same number of pixels(320×240 pixels) as the live view image so as to correspond to thenumber of pixels of the liquid crystal display monitor 13. However, theconfirmation image may be larger than the live view image.

Moreover, although the arrangement image is generated by resizing thepartial area, the arrangement image may be generated by extracting thepixel of the partial area according to the size of the correspondencearea.

Return to FIG. 6, after finishing a generation of the confirmationimage, the display 70 displays the generated confirmation image (stepS107).

Then, the input device 50 acquires the reconstruction setting togenerate the main image from the memory 34, the input device 32 or thelike (step S108). The reconstruction setting includes information suchas a set of a distance between the main lens 311 and the reconstructionsurface, and the part which a reconstruction image occupies on thereconstruction surface, a set of information on whether a filter isused, and filter setting, the number of pixels of the reconstructionimage, and the like. A part, where the input device 50 sets the distanceto a reconstruction surface (it is also referred to as a reconstructiondistance or a re-focal length) based on the information from the inputdevice 32 or the memory 34, and stores the distance in a buffer set inthe RAM 23, is also referred to as a distance setter. Moreover, sincethe reconstruction distance is a distance for focusing at the time ofreconstruction of the image, it is also referred to as the re-focallength.

When the reconstruction setting is acquired, the main image generator640 performs the main image generation process using the reconstructionsetting (step S109). In the present embodiment, the main imagegeneration process is the image generation process 3 described below.

The functional constitution of the main image generator 640 whichperforms the main image generation process will be described withreference to FIG. 13A to FIG. 13D. In the present embodiment, the mainimage generator 640 corresponds to the third reconstruction imagegenerator 730 illustrated in FIG. 13A.

The third reconstruction image generator 730 includes a sub-imageextractor 731, a pixel displacement degree calculator 732, a filterprocessor 733, an image displacement degree calculator 734, anarrangement interval determiner 734A, a partial area definer 735, apartial image extractor 735A, an image arranger 736, and an outputdevice 737. By means of such elements, the third reconstruction imagegenerator 730 generates a third reconstruction image (main image) as anoutput image from the light field image LFI.

The sub-image extractor 731 extracts the sub-images which form the lightfield image LFI as target sub-images in order, and transfers theextracted sub-images to the pixel displacement degree calculator 732.

The pixel displacement degree calculator 732 sets each pixel in thetarget sub-image as a target pixel in order, and calculates the degree(pixel displacement degree of the target pixel) indicating a degree ofdisplacement of the pixel corresponding to the target pixel in theperipheral sub-images. The pixel displacement degree corresponds todistance with the photographic object of the target pixel. Although thepixel displacement degree may be calculated using an arbitrary manner ofestimating the distance with the photographic object according to thepixel of the light field image LFI, the present embodiment calculatesthe pixel displacement degree using the functional constitution of FIG.13B in the following manner.

First, a target pixel selector 7321 selects a target sub pixel to beused as the processing object on the target sub-image extracted by thesub-image extractor 731. Then, a peripheral image selector 7322 selectsthe sub-image (peripheral image) to be used as a comparison objectlocated within a predetermined range from the target sub-image. In thisembodiment, the sub-images on the right of the target sub-image (on theleft when there is nothing on the right, in this case right-and-leftreversal) are selected as SR1, SR2 . . . SRk in a position order. k isthe natural number determined according to a setting. The peripheralimage selector 7322 is also referred to as a second selector, when thesub-image extractor 731 is referred to as the first selector.

Then, a comparison pixel selector 7323 selects the pixel (comparisonpixel) to be used as the comparison object on the peripheral image. Inthis embodiment, assuming that the coordinate of the target pixel on thetarget sub-image is represented by (x, y) and the current pixeldisplacement is represented by d, the pixel located in (x+d, y) on theperipheral image SR1, the pixel located in (x+2d, y) on the peripheralimage SR2, . . . , the pixel located in (x+kd, y) on the peripheralimage SRk are selected as the comparison pixels, respectively.

Next, a difference calculator 7324 calculates a sum of absolute valuesof differences between the pixel values of the target pixel and thecomparison pixel (pixel value of the pixel which is displaced by dpixels to the right). Such calculation is repeated within a range ofpossible parallax (value of d).

After that, a correspondence pixel determiner 7325 determines the valueof d of which the obtained sum of absolute values of differences is theminimum. As for the obtained value d, the pixel located in (x+d, y) onthe peripheral image SR1, the pixel located in (x+2d, y) on theperipheral image SR2, . . . , the pixel located in (x+kd, y) on theperipheral image SRk can be estimated as the pixels in which the samephotographic object as the target pixel is captured (correspondencepixels), respectively. A position displacement indicated by the value ofd is determined as the pixel displacement degree. The differencecalculator 7324 and the correspondence pixel determiner 7325 function asa searcher which searches the second corresponding to the target pixel(first pixel) in cooperation.

The pixel displacement degree calculator 732 calculates the pixeldisplacement degrees for all pixels included in the target sub-image,and transfers the pixel displacement degrees to the filter processor733. The filter processor 733 compares a distance to the photographicobject corresponding to each pixel indicated by the pixel displacementdegree and a distance to the reconstruction surface included in thereconstruction setting acquired from the input device 50. Then, ablurring addition is applied to each pixel of the light field image byfiltering according to the difference. When the difference between there-focal length and the distance to the photographic object is equal toor greater than a predetermined value, a blurring addition process isapplied in accordance with a blurring intensity which is set. Forexample, a Gaussian filter is used for the blurring addition process.The pixel displacement degree calculator 732 can be refereed to as a“pixel processor” which processes the target pixel and thecorrespondence pixel based on a position relationship between the targetpixel and the correspondence pixel.

When a certain pixel represents a point where the photographic objectexisting long way away from the camera is photographed, the pixeldisplacement degree (which corresponds to the amount of positiondisplacement) is small, and a certain pixel represents a point where thenear photographic object is photographed, the pixel displacement degreeis large. The relation of the position displacement and a distancebetween the photographic object and the camera in the real world dependson focal lengths of the main lens 311 and the micro lenses 312-i in themicro lens array 312, and the position on which the imaging element 313is arranged, and the size of the imaging element 313. The correspondencerelation of the position displacement and a photographic object distanceis obtained by experiments in advance, and is stored as a displacementdistance table in the ROM 22.

In the present embodiment, the filtering is performed using adisplacement distance table. The displacement distance table is obtainedby experiments in advance, and is stored in the ROM 22. The displacementdistance table stores a level (LV) indicating a distance in thepredetermined range, the displacement degrees (the image displacementdegree and the pixel displacement degree) belonging to the level, and adistance to the reconstruction surface belonging to the level,associated each other.

The filter processor 733 calculates a difference between the level towhich the pixel displacement degree belongs and the level to which thedistance to the reconstruction surface belongs using the displacementdistance table. The pixel whose difference is 0 is not applied thefiltering, under an estimation that it may correspond to thephotographic object sufficiently close to the reconstruction surface.That is, blurring is not added. On the other hand, when the differenceis equal to one or more, the blurring is added since the photographicobject for the pixel is displaced from the reconstruction surface. Thefiltering is performed so that stronger blurring is added, larger thedifference is.

The image displacement degree calculator 734 calculates the imagedisplacement degree of the target sub-image as is the case in thedisplacement value calculator 722 of FIG. 10A. The image displacementdegree calculator 734 calculates the image displacement degrees for botha horizontal direction and a vertical direction, and after that,averages both degrees to be the image displacement degree of the targetimage.

The arrangement interval determiner 734A determines an interval forarranging the image of the partial area on the reconstruction imageaccording to the image displacement degree. The arrangement interval isdetermined depending on the image displacement degree, the focal lengthof the main lens 311 included in the imaging setting information, thefocal length of micro lenses 312-i in the micro lens array 312, theposition on which the imaging element 313 is arranged, and the distanceto the reconstruction surface included in the reconstruction setting(reconstruction distance).

The difference of the angles of view for every micro lens is smaller asthe photographic object exists further away, and thus movement in thesub-image is also small. This is followed so that the arrangementinterval of the partial area extracted from the sub-image, for which theimage displacement degree is small, is made small. On the contrary, thedifference of the angles of view for every micro lens is larger as thephotographic object exists nearer, and thus the movement in thesub-image is also large. This is followed so that the arrangementinterval of the partial area extracted from the sub-image, for which theimage displacement degree is large, is made large.

In the present embodiment, the arrangement interval is determined usinga table (arrangement interval table) stored in the ROM 22, in which thearrangement interval is determined using the focal length of the mainlens 311, the distance to the reconstruction surface, and the imagedisplacement degree as indexes.

The arrangement interval table defines appropriate value of thearrangement interval obtained by experiments for each of values of theimage displacement degree, the distance (reconstruction distance) to thereconstruction surface, and the focal length of the main lens 311, inaccordance with the focal length of the micro lenses 312-i in the microlens array 312 which is a fixed parameter, and the position on which theimaging element 313 is arranged (for example, FIG. 13C). In the exampleof FIG. 13C, the arrangement interval is set to be larger as adifference between the level of the image displacement degree defined bythe above-described displacement distance table, and the level of thedistance to the reconstruction surface is larger.

The arrangement interval may be calculated according to an equationwhich is obtained by experiments in advance, and is stored in the ROM22, and in which the above-described parameter is used as a variable.

The partial area definer 735 acquires the partial area sizecorresponding to the image displacement degree from the area definitiontable of FIG. 10B, and corrects the size based on the blurring intensityof the target sub-image.

The blurring intensity of the target image is a degree for indicatingthe degree of displacement of the photographic object included in thetarget sub-image from the reconstruction surface, and for determiningwhether the part is to be blurred to display. In this embodiment, adifferent between the level of the image displacement degree defined bythe above-described displacement distance table, and the level of thedistance to the reconstruction surface is regarded as the blurringintensity.

One side of the partial area after correction is larger than thearrangement interval. Therefore, when the image of the partial area isarranged on the third reconstruction image, arrangement images areoverlapped. Since the pixel values are averaged for the overlapped part,the blurring is stronger as the number of the arrangement images to beoverlapped increases. Therefore, if the blurring intensity is strong,the partial area size is corrected so that the partial area size is madelarger. Since the partial area definer 735 determines the partial areasize, the partial area definer 735 also behaves as a “size determiner”.

The partial image extractor 735A extracts the pixels of the partial areaas the partial image as it is. The process in which the partial imageextractor 735A extracts the partial image in the size defined by thepartial area definer 735 is also expressed as a clipping process. Thepartial image extractor 735A is also referred to as a clipper whichclips the partial images from the sub-images.

The image arranger 736 arranges the image extracted by the partial imageextractor 735A on the generation image (third reconstruction image). Asa result, the image of the partial area is arranged on thereconstruction image without resizing. In this case, an interval withthe image of the partial area previously arranged is in accordance withthe arrangement interval calculated by the above-described imagedisplacement degree calculator 734. In this case, the arrangementinterval and the partial area size are set so that the arranged imagesoverlap. As for the overlapping part, the pixel values of the pixelsarranged and overlapped are averaged to be the pixel value of thegeneration image. As a result, stronger blurring is added as theoverlapping part is larger. The sub-image extractor 731 through theimage arranger 736 perform the above-described process for allsub-images included in the light field image LFI to generate the thirdreconstruction image.

Then, the output device 737 outputs the generated third reconstructionimage to outside (to the display 35, the memory 34, removable media 38and the like), and whereby storing the third reconstruction image as theimage for viewing.

Next, the process (image generation process 3) performed by the thirdreconstruction image generator 730 will be described with reference toFIG. 14 and FIG. 15.

When arriving step S109 of the image output process 1 (FIG. 6), the mainimage generator 640 (the third reconstruction image generator 730)starts the image generation process 3 illustrated in FIG. 14.

In the image generation process 3, k3 represents a counter variablefirst, and the sub-image extractor 731 extracts k3-th sub-image in thelight field image LFI as a target sub-image (step S401).

Next, for each pixel of the target sub-image, the pixel displacementdegree calculator 732 calculates the pixel displacement degree asdescribed above (step S402).

For each pixel in the target sub-image, the filter processor 733calculates the difference (level difference) between the pixeldisplacement degree and the reconstruction distance using thedisplacement distance table (step S403).

Next, the filter processor 733 applies the filtering process to thepixel, of which the difference with the reconstruction distance is equalto or greater than a predetermined value (the level difference is not0), according to the level difference under a condition stored by theROM 22 (step S404). The filter processor 733 performs a Gaussianfiltering using larger value of δ as the level difference is larger, forexample, the kernel of 3×3 is used for the pixel of which the leveldifference is 1, and the kernel of 9×9 is used for the pixel of whichthe level difference is 2.

Next, the image displacement degree calculator 734 acquires (obtains)the image displacement degree of the target sub-image (step S405). Atstep S405, the image displacement degree is acquired for the sub-imageon the right as is the case in step S302 of FIG. 11, the imagedisplacement degree is further acquired for the sub-image on lower side(upper side when there is nothing on the lower side), and an average ofboth degrees is regarded as the displacement degree of the targetsub-image.

Since the image displacement degree is calculated based on the positiondisplacements in the vertical direction and the horizontal direction inexchange for increasing calculation amount, the image displacementdegree which incorporates the photographic object distance of thesub-image more correctly in comparison with a case of step S302 of FIG.11, and the image quality of the generation image improves. In addition,as a manner for improving the accuracy of the image displacement degree,it is supposed that the image displacement degrees may be calculated formore sub-images in one direction.

The image displacement degree may be calculated by averaging the pixeldisplacement degrees for respective pixels included in the targetsub-image calculated at step S402.

Next, at step S406, the arrangement interval determiner 734A retrievesthe numerical value of the arrangement interval in accordance with theimage displacement degree, the focal length of the main lens 311, andthe distance to the reconstruction surface from the arrangement intervaltable (for example, FIG. 13C) stored in the ROM 22, and determines(sets) the retrieved numerical value as the arrangement interval.

Next, the partial area definer 735 acquires the partial area size(L2×L2) corresponding to the image displacement degree from the areadefinition table (FIG. 10B). Furthermore, the size is corrected so thatthe partial area size is larger as the blurring intensity is stronger(for example, as in FIG. 13D) to determine the partial area size(L2×L2). Then, the square are with L2×L2 at the central part of thetarget sub-image is defined as the partial area (step S407). L2 islarger than the arrangement interval.

In the example of FIG. 15, since the image displacement degree betweenS₁₁ and S₁₂ is large, the partial area is large. On the other hand, theimage displacement degree for S₂₃ is small, and therefore the partialarea is small.

The image of the partial area is arranged on the third reconstructionimage which is the generation image at the arrangement intervaldetermined at step S406 (FIG. 15 lower side, step S408). Sincearrangement images overlaps, the overlapped part is averaged todetermine the pixel value. In the lower side of FIG. 15, the image (areawith horizontal lines) of the partial area for S₁₁ and the image (areawith vertical lines) of the partial area for S₁₂ are overlapped.

After the image is arranged to the correspondence area for the currenttarget sub-image at step S408, it is next determined whether theabove-described process has been performed for all sub-images (stepS409). When an unprocessed sub-image exists (step S409; NO), the countervariable k3 is incremented (step S410), and the process is repeated fromstep S401 for a following target sub-image.

On the other hand, when the above-described process has been performedfor all sub-images (step S409; YES), the completed third reconstructionimage is regarded as the main image, and the image generation process 3is terminated.

Returns to FIG. 6, after finishing a generation of the main image, thegenerated main image is stored in the memory 80 (step S110), and theimage output process 1 is terminated.

The process which arranges the partial images on the reconstructionimage RI will be described using a setting example in which one side ofthe partial image is set to the twice of the arrangement interval andusing FIGS. 16A and B. The four partial images overlap in each partexcept for the edge parts of the reconstruction image. First, thesub-image S₁₁ at the upper left of the light field image LFI is selected(step S401). A partial area (partial image PI₁₁) is clipped (extracted)from the selected sub-image S₁₁ (step S407), and this partial image PI₁₁is arranged at the upper left of the reconstruction image RI (S408).Return to step S401, the adjacent sub-image S₁₂ on the right is selected(step S401). A partial image PI₁₂ is clipped from the selected sub-imageS₁₂ (S407), and the partial image PI₁₂ is arranged at the right of thepartial image which has been arranged in the reconstruction image RI(S408). Since one side of the clipped partial image PI₁₂ is twice thearrangement interval, it is arranged with an overlap in horizontaldirection. The pixel values of the overlapped part are added on abuffer. Similarly, a partial image PI₁₃ is clipped from adjacentsub-image S₁₃ on the right of the sub-image S₁₂, it is arranged with anoverlap in horizontal direction (FIG. 16A).

Such operations are repeated to the right end of the light field imageLFI. When the process for the sub-image S_(IN) at the right end iscompleted, it returns to the left end of the light field image LFI, andselects the sub-image S₂₁ on the second row from the top (step S401). Apartial image PI₂₁ clipped from the selected sub-image S₂₁ is arrangedunder the partial image P11 which has been first arranged in thereconstruction image RI (step S408). Since one side of the clippedpartial image PI₂₁ is twice the arrangement interval, as illustrated inFIG. 16B, the arrangement is made with an overlap in a verticaldirection. The pixel values of the overlapped part are added on abuffer.

Such operations are repeated to the right end of the light field imageLFI, as is the case of the first row. When the process for allsub-images S₂₁ to S₂N in the second row is completed, the sub-image S onthe following row in the light field image LFI will be processed. Theseoperations are repeated. When it is determined that the process for thesub-image S_(MN) at the lowermost and rightmost of light field image LFIis completed (step S409; Yes), after normalizing pixel values of theoverlapped part as appropriate, the created reconstruction image RI isdisplayed on the display 35 or output from the output device 33, and isstored in the memory 80 (removable media 38 and the like) (S110).

In the image generation process 3, blurring processing for everysub-image and blurring processing for every pixel are applied,respectively. For that reason, even when the error is included inphotographic object distance estimation for every pixel, it is averagedby the blurring processing (overwriting) for every sub-image, and theinfluence of the error is eased. Therefore, it is possible to generatethe main image having high image quality.

As having described above, the digital camera 1 of the presentembodiment includes a plurality of reconstruction image generation meanssuitable for securable calculation time and a usage. Therefore,appropriate reconstruction images suitable for various usages, such asthe live view, confirmation after photographing, and for viewing, can begenerated and displayed at a preferred speed. In other words, thereconstruction images can be generated with a suitable generation methodaccording to a purpose of using an image. Thus, the digital camera 1 ofthe present embodiment can display necessary and sufficientreconstruction images with required and sufficient quality at sufficientspeed according to a status, and whereby improving convenience for auser.

Specifically, according to the first reconstruction image generationprocess, it is possible to generates the reconstruction image (the firstreconstruction image) including the contents of all sub-images includedin the light field image LFI with small calculation amount.

Therefore, even when sufficient calculation amount is not securable,such as a case in which the live view is used, the reconstruction imagewhich can be used for photographing preparation can be generatedespecially at high speed, and can be displayed. Moreover, in comparisonwith a case of using the image in which pixels are extracted from thesub-image one by one and arranged, an image can be generated with higherquality.

According to the second reconstruction image generation process, it ispossible to generate the reconstruction image (the second reconstructionimage) of which the image quality is higher than the image quality ofthe first reconstruction image using more calculations than firstreconstruction image generation process, in accordance with the degreecorresponding to a distance to the photographic object in sub-imageunit. Therefore, the reconstruction image of which the image quality ishigher than the image quality of the first reconstruction image can bedisplayed at high speed for the confirmation after photographing.

According to the third reconstruction image generation process, it ispossible to generate the reconstruction image (the third reconstructionimage) with high image quality using more calculations than secondreconstruction image generation process, the image being applied theblurring addition process in sub-image unit and the blurring additionprocess in pixel unit. Therefore, the reconstruction image with highimage quality cab be generated at a terminal, such as a digital camera,which has limited capability of processing speed.

Moreover, by arranging images while inverting every sub-image in fourdirections, the reconstruction image arranged correctly can be generatedwith small calculation amount from the image which is photographed whilesub-images are inverted due to the configuration of the optical system.

The digital camera 1 of the present embodiment is designed to define alarger partial area for a sub-image with a large image displacementdegree so that the information on overall partial area appears in thereconstruction image.

A large image displacement degree represents that the photographicobject of the target sub-image exists in a near position from the mainlens, and in the adjacent sub-image, the visible position is displacedlargely due to the parallax of the corresponding micro lens. In thisembodiment, the photographic object at the central part is regarded as aphotographic object on behalf of the sub-image.

A large displacement of corresponding photographic object between thesub-images means that the necessary size of partial area to avoid anomission of information, when forming the overall image by connectingthe partial areas of adjacent sub-images, is large. In the presentembodiment, the partial area is made larger as the image displacementdegree is larger, and the partial area is arranged to the correspondencearea of the reconstruction image, whereby preventing the omission ofinformation in the generation image. Moreover, the partial area is madesmaller as the image displacement degree is smaller, so that overlap ofinformation which is presented in a section between the adjacentsub-images does not appear too much on the reconstruction image.

According to the digital camera 1 of the present embodiment, even if thereconstruction image is generated at high speed (generated by processusing small calculations) by such a configuration, omission ofinformation and the degree of overlap of information is reduced takingthe position displacement by the parallax into consideration.

Moreover, since the blurring addition by an overwrite and the blurringaddition by a filtering after a photographic object distance estimationare used together in the main image generation process, the degree ofblurring (bokeh's flavor) can be changed by adjusting intensities ofboth blurring.

In each image generation process, the reconstruction distance may be seton the basis of manual operation (user's operation). Alternatively, thereconstruction distance may be set by measuring a distance to thephotographic object (for example, the photographic object of the centralpart, or the photographic object designated by the user's operation)which is desired to focusing a phase difference sensor or the like, andfocusing on the photographic object. Furthermore, again the re-focallength may be set by reselecting a photographic object which is desiredto focusing a touch panel or the like after displaying thereconstruction image.

(Embodiment 2)

The embodiment 2 of the present invention will be described.

The digital camera 1 of the present embodiment has a physicalconfiguration illustrated in FIG. 2. As for the digital camera 1 of thepresent embodiment, in comparison with the corresponding elements in theembodiment 1, the CPU 21 can process at higher speed, and the display 35has higher resolution, for example, VGA (640×480 pixels).

Other elements of FIG. 2 are the same as the elements of the digitalcamera 1 according to the embodiment 1.

In the digital camera 1 of the present embodiment, since the display 35is large, the impreciseness of an image is noticeable. Then, utilizingthat the CPU 21 can process at higher speed, a reconstruction image withhigh preciseness and high image quality for displaying on the display 35of the digital camera 1 is generated.

The digital camera 1 of the present embodiment includes functionalconstitution illustrated in FIG. 5.

In the present embodiment, the live view image generator 620 is thesecond reconstruction image generator 720, the confirmation imagegenerator 630 is the third reconstruction image generator 730, and themain image generator 640 is a fourth reconstruction image generator.Other elements of FIG. 5 are the same as the corresponding elements ofthe digital camera 1 according to the embodiment 1.

The second reconstruction image generator 720 and the thirdreconstruction image generator 730 have the same configuration as theelements with same names in the embodiment 1.

The fourth reconstruction image generator performs a reconstructionimage generation process described below to generate the reconstructionimage from the light field image LFI, and transfers the generated imageto the memory 80.

The process performed by the digital camera 1 of the present embodimentwill be described with reference to a flowchart.

When the digital camera 1 is powered on, and the input device 50receives an operation for preparing a photographing, the digital camera1 starts an image output process 2 illustrated in FIG. 17.

In the image output process 2, process from step S501 to step S502 issimilarly performed with step S101 to step S102 of the image outputprocess of FIG. 6.

In the present embodiment, the image generation process 2 (FIG. 11) isperformed as is the case in the embodiment 1 at step S503 to generatethe live view image.

After generating the live view image at step S503, the digital camera 1performs the process from step S504 to step S505 similarly with stepS104 to step S105 of the image output process 1 of FIG. 6.

At step S506, the confirmation image generator 630 which is the thirdreconstruction image generator 730 of FIG. 13A performs the imagegeneration process 3 illustrated in FIG. 14 as in the embodiment 1 togenerate the confirmation image.

After generating the confirmation image at step S506, the digital camera1 performs step S507 similarly with step S107 of the image outputprocess of FIG. 6.

At step S508, the main image generator 640 which is the fourthreconstruction image generator performs an image generation process 4 togenerate the main image.

The image generation process 4 reconstructs an image by performingweighted addition on the pixel values of the light field image for everymicro lens through which the light has passed.

The image generation process 4 generates the reconstruction image in thefollowing procedures.

-   (1) Specifying a position on the micro lens array where a light beam    from a target pixel to reconstruct has reached passing through a    principal point of the main lens.-   (2) Calculating a main lens blur area (area on the micro lens array    which the light beam from the target pixel reaches) with a radius    based on the re-focal length corresponding to the reconstruction    surface centering on the specified position.-   (3) Specifying the micro lenses a part of or all of which are    included in the main lens blur area among the micro lenses included    in the micro lens array.-   (4) Selecting one of the specified micro lenses.-   (5) Calculating an area of overlapping section of the selected micro    lens and the main lens blur area, and dividing the result by an area    of the micro lens to obtain a weighting factor.-   (6) Obtaining a pixel value on the sub-image located in a position    on which the light beam from the target pixel forms image by the    selected micro lens.-   (7) Multiplying the acquired pixel value by the weighting factor to    obtain a corrected pixel value.-   (8) Calculating the corrected pixel values for all of micro lenses,    a part of or all of which are included in the main lens blur area,    and sums the corrected pixel values.-   (9) Dividing the total of the corrected pixel values by the total of    the overlap area to obtain the target pixel value of the image to    reconstruct.

After generating the main image at step S508, the digital camera 1performs step S509 similarly with step S110 of the image output processof FIG. 6, and terminates the image output process 2.

As having described above, the digital camera 1 of the presentembodiment can display the live view image with image quality equivalentto the confirmation image of the embodiment 1. Moreover, theconfirmation image can be displayed at image quality equivalent to themain image of the embodiment 1.

As a modification of the present embodiment, the live view image may begenerated using the image generation process 1 instead of the imagegeneration process 2.

According to such configuration, the live view image which desires thefastest generation speed can be generated with a small calculationamount as is the case in the embodiment 1, whereas each of theconfirmation image and the main image has image quality higher than theimage quality in the embodiment 1.

The fourth reconstruction image generator of the present embodiment maygenerate the reconstruction image using a known arbitrary process whichgenerates the reconstruction image from the light field image LFI.

(Embodiment 3)

The embodiment 3 of the present invention will be described.

The digital camera 1 of the present embodiment has a physicalconfiguration illustrated in FIG. 2. As for the digital camera 1 of thepresent embodiment, in comparison with the corresponding element in theembodiment 1, the CPU 21 can process at higher speed. Other elements ofFIG. 2 are the same as the elements of the digital camera 1 according tothe embodiment 2.

In the embodiment 1 and the embodiment 2, an image which does notinclude blurring is displayed as the live view image. The embodiment 3is characterized by displaying an image to which blurring is added froma phase of the live view image (prior to the photographing), utilizingthat the CPU 21 can process at higher speed.

The digital camera 1 of the present embodiment includes the functionalconstitution illustrated in FIG. 5.

In the present embodiment, the live view image generator 620 is a fifthreconstruction image generator 740 illustrated in FIG. 18. Otherelements of FIG. 5 are the same as the corresponding elements of thedigital camera 1 according to the embodiment 2.

The fifth reconstruction image generator 740 is different from the thirdreconstruction image generator 730 illustrated in FIG. 13A in that:

(1) the fifth reconstruction image generator 740 does not includeelements corresponding to the pixel displacement degree calculator 732and the filter processor 733; and

(2) the image displacement degree calculator 734 a acquires the pixeldisplacement degrees for the sub-images in a predetermined area. Theaverage value is used as a displacement degree for entire sub-images todetermine an arrangement area size and an arrangement interval.

Other elements are the same as the elements with same names of the thirdreconstruction image generator 730 of the embodiment 1.

The process performed by the digital camera 1 of the present embodimentwill be described with reference to a flowchart.

When the digital camera 1 is powered on, and the input device 50receives an operation for preparing a photographing, the digital camera1 starts an image output process 3 illustrated in FIG. 19.

In the image output process 3, process from step S601 to step S602 issimilarly performed with step S101 to step S102 of the image outputprocess of FIG. 6 according to the embodiment 1.

In the present embodiment, the image generation process 5 (FIG. 20) isperformed at step S603 to generate the live view image.

The image generation process 5 will be described with reference to FIG.20.

In the image generation process 5, the image displacement degreecalculator 734 a first acquires one or more image displacement degreesfor a part of sub-images determined by a setting among the sub-imagesincluded in the light field image LFI (step S701). Here, “a part of”sub-images is set so that the degree of displacement of overall lightfield image LFI is presented in the image displacement degree obtainedfrom “a part of sub-images.” Specifically, a part of sub-images may besub-images of a predetermined central part of the light field image LFI,or four sub-images of corners, or the like.

The manner of acquiring the image displacement degree for each sub-imageis the same as step S302 of the image generation process 2 (FIG. 11)according to the embodiment 1.

Next, the image displacement degree of the overall light field image LFIis determined based on the image displacement degrees of a part ofsub-images calculated at step S701. Here, a plurality of calculatedimage displacement degrees are averaged, and the result is set as thedisplacement degree for the overall sub-images included in the lightfield image LFI (step S702).

Then, based on the displacement degree for the overall sub-images, thearrangement interval and partial area size which are applied to allsub-images are calculated in accordance with the conditions stored inadvance at the ROM 22 or the memory 34 (step S703). As the displacementdegree is larger, the partial area and the arrangement interval is alsolarger. Moreover, the partial area size is larger than the arrangementinterval. In this embodiment, when the displacement degree of theoverall sub-images is 10, for example, the arrangement interval is 10pixels, and the partial area size is 20 pixels×20 pixels.

Next, k4 represents a counter variable, and the sub-image extractor 731extracts the k4-th sub-image in the light field image LFI as a targetsub-image (step S704).

Then, the partial area definer 735 defines the partial area in the sizecalculated at step S703 at the central part of the target sub-image asis the case for step S407 of FIG. 14 (step S705). As identifying theportion (partial area) of the sub-image which is either inside oroutside a region (clipping partial area from sub-image), the partialarea definer 735 also behave as a “clipper.”

Furthermore, the image arranger 736 arranges the images of the partialarea at the arrangement interval calculated at step S703 as is the casefor step S408 of FIG. 14 (step S706). At this time, the images arearranged while inverting in four directions. At step S703, since thepartial area size is larger than the arrangement interval, the partialimages are arranged so as to overlap, as illustrated in FIG. 15.Averaging the pixel values of the overlapping section causes an additionof blurring to the generated live view image in accordance with theposition displacement degree (which corresponds to the photographicobject distance of the overall photographic object) calculated at stepS702.

At step S707, it is determined whether the process which arranges imageshas been performed for all sub-images.

When an unprocessed sub-image remains, (step S707; NO), k4 isincremented (step S705), and the process is repeated from step S704.

On the other hand, when the process has been performed for allsub-images (step S707; YES), the generated image is output as the liveview image, and the image generation process 5 is terminated.

Return to FIG. 19, the process from step S604 to step S609 is similarlyperformed with step S504 to step S509 of FIG. 17 in the embodiment 2,and this process is terminated.

As having described above, according to the digital camera 1 of thepresent embodiment, the image to which the blurring is added isdisplayed prior to the photographing (in the live view mode to generatethe live view image). Accordingly, the user can expect a completed imageeasily prior to the photographing.

Moreover, in the live view mode to generate the live view image it isnot needed to calculate the displacement degree for every pixel, andthere is a little increase of the necessary calculation amount foradding blurring. Therefore, the image to which blurring is added can begenerated at high speed.

(Embodiment 4)

The embodiment 4 of the present invention will be described.

The digital camera 1 of the present embodiment has a physicalconfiguration illustrated in FIG. 2. The function of each element of thedigital camera 1 is the same as the function in the digital camera 1according to the embodiment 1.

According to the embodiment 1, the image displacement degrees arecalculated for all sub-images in the image generation process 2 and theimage generation process 3. In the image generation process 5 of theembodiment 3, the image displacement degrees of a part of sub-images(the predetermined part of the central part or four sub-images ofcorners, or the like) are calculated, and its average is set as theoverall image displacement degree. On the other hand, in the presentembodiment, when generating the live view image, the image displacementdegrees are calculated for a part of sub-images, and from thecalculation result, the image displacement degrees of other sub-images(other than the object for calculating the image displacement degree)are estimated. Moreover, the present embodiment is characterized byselecting the sub-images used as the object for calculation in cyclicmanner.

The digital camera 1 of the present embodiment includes the functionalconstitution illustrated in FIG. 5.

In the present embodiment, the live view image generator 620 is a sixthreconstruction image generator 750 (FIG. 21), and the confirmation imagegenerator 630 is the third reconstruction image generator 730illustrated in FIG. 13A. The main image generator 640 is the fourthreconstruction image generator described in the embodiment 2. Otherelements of FIG. 5 are the same as the elements of the digital camera 1according to the embodiment 1.

The sixth reconstruction image generator 750 includes, as illustrated inFIG. 21, an image displacement degree calculator 751, an imagedisplacement degree estimator 752, the sub-image extractor 721, thepartial area definer 723, the arrangement image generator 724, the imagearranger 725, and the output device 726. The sixth reconstruction imagegenerator 750 includes the image displacement degree calculator 751 andthe image displacement degree estimation unit 752 instead of thedisplacement value calculator 722 in comparison with the secondreconstruction image generator 720 of FIG. 10A. Other configurations arethe same as the second reconstruction image generator 720.

The image displacement degree calculator 751 selects the sub-image(calculation image) which is a calculation object for the displacementdegree whenever the live view image is generated, in cyclic manner.Then, the image displacement degree is calculated for the selectedcalculation image. Specifically, the sub-image is divided into n groups(for example, n=2). When the live view images are sequentiallygenerated, a group is selected in order for every loop as a target group(group including a calculation image). The sub-images belonging to thetarget group are used as the calculation image. The value of n is set inadvance and stored in the memory 34. The calculation manner of the imagedisplacement degree is the same as the manner by the displacement valuecalculator 722 in the embodiment 1.

The image displacement degree estimation unit 752 estimates the imagedisplacement degree of sub-image (estimation image) other than thecalculation image using the image displacement degree of the calculationimage calculated by the image displacement degree calculator 751. Thespecific manner of the estimation is described below.

The process performed by the digital camera 1 of the present embodimentwill be described with reference to FIG. 22. The digital camera 1 of thepresent embodiment is powered on, and the input device 50 receives anoperation for preparing a photographing, the digital camera 1 starts animage output process 4 illustrated in FIG. 22.

In the image output process 4, step S801 and step S802 are similarlyperformed with step S101 and step S102 of the image output process 1(FIG. 6) performed in the embodiment 1.

The image displacement degree calculator 751 of the live view imagegenerator 620 (the sixth reconstruction image generator 750) updates aframe flag (step S803). The frame flag is a flag for selecting a groupfor calculating the calculation image. The frame flag is set as 1 atfirst, and is incremented by 1 for every loop (step S801 to step S807)in which the live view image is generated. When the frame flag exceedsthe number of groups n which is set, the frame flag returns to 1. As aresult, in the case of n=4 for example, the frame flag is set thenumerical values of 1 to 4 in cyclic manner such as 1, 2, 3, 4, 1, 2, 3,4, and 1- - - .

After updating the frame flag at step S803, the image displacementdegree calculator 751 calculates the image displacement degree of thecalculation image, and starts the process (displacement degreeestimation process) for estimating the image displacement degree of theestimation image.

The displacement degree estimation process performed at step S804 willbe described with reference to FIG. 23. In the displacement degreeestimation process, the image displacement degree calculator 751 firstselects the calculation image according to the current frame flag (stepS901).

The selection manner of the calculation image selected in this step willbe described using FIG. 24 and FIG. 25. In FIG. 24 and FIG. 25, eachsub-image is represented in square form. For example, when the sub-imageis divided into two groups (n=2), any of the sub-image illustrated inblack in FIG. 24 and the sub-image illustrated in white is selected asthe calculation image. For example, the sub-images in black is selectedwhen the frame flag is 1, and the sub-images in white is selected whenthe frame flag is 2.

Alternatively, the light field image LFI is divided into n areas each ofwhich includes the sub-images, and a calculation image is selected incyclic manner in the divided area. For example, when n=9, the lightfield image LFI is divided into the area (thick line) which includessub-images with three lines in vertical direction and with three linesin horizontal direction, as illustrated in FIG. 25. Then, the sub-imagesin the area are numbered from 1 to 9. For example, the sub-images a1 toa9 in FIG. 25 mean the first to ninth sub-images in a first area (“a”area), respectively. One sub-image with the number which matches withthe current frame flag is selected as the calculation image from eacharea (in the example of FIG. 25 “a” area to “1” area).

After selecting the calculation image at step S901, the imagedisplacement degree calculator 751 next selects one target sub-imagefrom calculation image (step S902). The image displacement degreecalculator 751 calculates the image displacement degree for the targetsub-image (step S903). The calculation manner of the image displacementdegree is the same as the manner in the embodiment 1.

The image displacement degree calculator 751 determines whether or notthe amount of change of the image displacement degree for the targetsub-image is equal to or larger than a threshold (step S904). In thisstep, the image displacement degree calculator 751 compares, for thetarget sub-image, a difference between the image displacement degreeestimated in the last displacement degree estimation process and theimage displacement degree calculated in current process, with apredetermined threshold. The threshold is obtained by experiments inadvance, and is stored in the memory 34. In the case where the numericalvalues of n is relatively small and time lag from the time calculatedlast time (for example, when n=2) and the like, the difference betweenthe image displacement degree calculated in the last process and theimage displacement degree calculated in current process may be comparedwith a predetermined threshold.

When the amount of change is less than the threshold as a result of thecomparison (step S904; NO), the image displacement degree calculator 751sets a change flag of the target sub-image to OFF (step S905). On theother hand, when the amount of change is equal to or larger than thethreshold (step S904; YES), the image displacement degree calculator 751sets the change flag of the target sub-image to ON (step S906). Thechange flag is a binary variable associated with each sub-image.

After setting the change flag, the image displacement degree calculator751 next determines whether the image displacement degrees have beencalculated for all calculation images selected at step S901 (step S907).When an unprocessed calculation image exists (step S907; NO), theprocess is repeated from step S902 for the following calculation image.On the other hand, when all the calculation images has been processed(step S907; YES), process goes to nest step S908.

At step S908, one sub-image which has not been applied an estimationprocess is selected as a target image among images (estimation image)other than the calculation image (step S908).

The change flags for peripheral calculation images (peripheral image)are checked to determine whether or not the image displacement degreechanged (step S909). For example, in the case of n=2, when there are apredetermined number of (for example, two or more) images of which thechange flags are ON, among the calculation images adjacent to the targetimage in four directions, it is determined that a peripheral image hasbeen change (step S909; YES). In this case, when all images of which thechange flags are ON change in the same direction (for example, adirection of increasing displacement degree), it may be determined thatthe change can be trusted and there is a change. Moreover, when theimages change in the different directions, it may be determined that thechange cannot be trusted and there is no change. When n is larger than2, m pieces of calculation images (for example, m=4) are selected inorder from an image close to the target image, and the change flags aresimilarly referred to.

When it is determined that there is a change (step S909; YES), the imagedisplacement degree of the nearest calculation image is estimated to bethe image displacement degree of the target image (step S910).Alternatively, the numerical value obtained by averaging the imagedisplacement degrees of the calculation images referred to at step S909may be set as an estimated value.

On the other hand, when it is determined that there is no change inperipheral images (step S909; NO), step S910 is skipped. Then, it isdetermined whether the above-described process has been performed on allestimation images (step S911). When an unprocessed estimation imageexists, the process from step S908 is repeated for the followingestimation image (step S911; NO). On the other hand, when all estimationimages have been processed (step S911; YES), the displacement degreeestimation process is terminated.

The process has been described above in which, when it is determinedthat the peripheral image has been changed, the image displacementdegree of the peripheral calculation image is estimated as the imagedisplacement degree of the estimation image. As another estimationmanner, a method may be adopted in which the amount of change of theperipheral calculation image is distributed according to the distancefrom the calculation image. FIG. 26 illustrates a concrete example ofthe distribution. FIG. 26 illustrates an example of a case where thelatest calculated value is different from the last calculated value (orlast estimated value), for the calculation image depicted by a thickline (+3 for the calculation image on left side and +1 for thecalculation image on right side). The difference of the imagedisplacement degrees is multiplied by a weight which is smaller asincreasing distance from the calculation image, the result ofmultiplication is added to the image displacement degree set in the lastloop process for the estimation image (square by dotted lines), and theresult of addition is regarded as the estimation result.

In FIG. 26, an arrow illustrates a direction of distribution (addition),and the numerical value on the arrow illustrates the number ofdistribution, respectively. In this example, two thirds of thedifferences of the calculation image is distributed to the estimationimages adjacent to the calculation image in four directions andobliquely adjacent to the calculation image, and one thirds of thedifferences is distributed to the estimation image next to that image.

As another estimation manner, the image displacement degree of theestimation image may be set by a three-dimensional interpolation processusing the image displacement degree of the calculation image.

Return to FIG. 22, after setting the image displacement degrees for allsub-images at step S804, the sub-image extractor 721 starts the imagegeneration process 2 illustrated in FIG. 11 using the set imagedisplacement degrees (step S805). In the image generation process 2,each step is performed as is the case in the embodiment 1 except foracquiring the image degree set in the image displacement degreeestimation process at step S302.

When the live view image is generated in the image generation process 2,the display 70 displays the generated image (step S806). Then, it isdetermined whether the input device 50 detects an imaging operation(step S807).

When the imaging operation is not detected according to thedetermination (step S807; NO), it returns to step S801 and a generationof the live view image is continued. On the other hand, when the imagingoperation is detected (step S807; YES), the confirmation image generator630 (the third reconstruction image generator 730) generates theconfirmation image by the image generation process 3 of FIG. 14 (stepS808). In this case, the image generation process 3 may calculate theimage displacement degrees for all sub-images, or may use the imagedisplacement degree set in the last displacement degree estimationprocess (step S804). Alternatively, the estimation process may beperformed by dividing the sub-image into the calculation image and theestimation image, as is the case in the process which generates the liveview image. At this time, a rate of the calculation image to the overallsub-image is set larger than a case in the live view image, wherebyallowing an improvement of estimation accuracy.

After that, the generated confirmation image is displayed (step S809),and the image generation process 4 generates the main image as is thecase in the embodiment 3 (step S810). Then, the generated main image isstored (step S811), and the image output process 4 is terminated.

As having described above, according to the digital camera 1 of thepresent embodiment, there is no necessity of calculating the imagedisplacement degree for all images, in the process which requires theimage displacement degree of each sub-image such as the image generationprocess 2. When generating a high quality image, since a calculation ofthe image displacement degree which requires large calculation amountcan be restricted, necessary calculation amount is reduced.

As a result, electric power required for calculation is also reduced.Accordingly, the drive time and the maximum number of photographing ofthe digital camera 1 can be increased.

Moreover, when generating the live view image, the calculation imagesare selected in cyclic manner. In the case where the calculation imageis fixed, when the photographic object of the calculation image isdifferent from other photographic objects of the estimation image, andwhen the noise has occurred in the calculation image, the calculationerror due to these matters always appear on the whole image. Accordingto the present embodiment, since the calculation image is changed forevery generation process, it is possible to ease the influence of thecalculation error even if some sub-images have error. Furthermore, inthe long term view, since the image displacement degrees are calculatedfor all sub-images, the live view images represent the overallphotographic object.

In the present embodiment, the live view image generator 620 generatesthe live view image by the image generation process 2 using the imagedisplacement degree calculated or estimated by the displacement degreeestimation process of FIG. 23. It is not limited to this manner, thedisplacement degree estimation process is applicable to an arbitraryprocess (the image generation process 3, image generation process 5, orthe like) which generates the reconstruction image using the imagedisplacement degree. For example, the live view image generator 620 maygenerates the image by the displacement degree estimation process andthe image generation process 3. Moreover, not only for the live viewimage generator 620 but also for the confirmation image generator 630 orthe main image generator 640, the displacement degree estimation processof the present embodiment can be used, when adopting a configurationwhich generates the reconstruction image using the image displacementdegree.

(Embodiment 5)

Hereinafter, the embodiment 5 of the present invention will be describedbased on drawings. The digital camera according to the embodiment 5 ischaracterized in that an image generation process 6 described below isused instead of the process performed by the main image generator 630 ofthe digital camera 1 in the embodiment 1. Other functional constitutionsand processes are the same as the functional constitutions and processesof the digital camera 1 according to the embodiment 1.

The digital camera 1 of the present embodiment has functionalconstitution illustrated in FIG. 5.

In the present embodiment, the live view image generator 620 is thesecond reconstruction image generator 720, the confirmation imagegenerator 630 is the third reconstruction image generator 730, and themain image generator 640 is a seventh reconstruction image generator760. Other elements of FIG. 5 are the same as the corresponding elementsof the digital camera 1 according to the embodiment 1.

The configuration of the seventh reconstruction image generator 760 willbe described with reference to FIG. 27A. The seventh reconstructionimage generator 760 includes, as illustrated in FIG. 27A, a clippingposition setter 761, a sub-image extractor 762, a partial area definer763, a partial image extractor 764, an image arranger 765, a combiner766, and an output device 767.

By means of such configuration, the seventh reconstruction imagegenerator 760 generates a predetermined number of intermediate images(for example, intermediate images ImI₁ to ImI_(n)), while changing anextracting (clipping) position of the partial area (partial image) fromthe sub-image. Then, respective generated intermediate images arecombined to generate the reconstruction image (main image).

The clipping position setter 761 sets the position where the partialarea (partial image) is extracted from the sub-image (target imageextracted by the sub-image extractor 762). In the present embodiment,different positions of partial areas are set for each generation processof intermediate image (for example, the central part of the target imagewhen generating the intermediate image ImI₁, and predetermined area atthe upper right of the target image when generating the intermediateimage ImI₂, and the like).

The sub-image extractor 762 extracts one sub-image from sub-images asthe target image for generating the intermediate image, as is the caseof the process performed by the sub-image extractor 731 for generatingthe reconstruction image of the embodiment 2,

The partial area definer 763 defines the partial area at a position onthe target image set by the clipping position setter 761, the size(clipping size) of partial area being determined in accordance with thefocal lengths of the main lens and the micro lenses of the micro lensarray, the position on which the imaging element 313 is arranged, andthe reconstruction distance. Since the focal lengths of the micro lensesare usually constant, the partial area is determined using a table (forexample, FIG. 27B) in which the focal length of the main lens 311, thereconstruction distance, and clipping size are associated with eachother, and which is stored in the ROM 22.

The partial image extractor 764 resizes the image of the partial area toa predetermined image size (for example, 4×4 pixels), and extractsresized image as the partial image. As identifying the portion (partialarea) of the sub-image which is either inside or outside a region(clipping partial area from sub-image) and extracting pixel data on thepartial area, the partial area definer 763 and the partial imageextractor 764 also behave as a “clipper” as a whole. The image arranger765 arranges the image extracted by the partial image extractor 764according to the position of the original sub-image to generate theintermediate image. For this reason, the image arranger 765 is alsoreferred to as an intermediate image generator. The specific example ofthe process is described below.

Then, the combiner 766 combines respective intermediate images togenerate the reconstruction image. The combiner 766 may generate thereconstruction image using an arbitrary known technique which summarizesa plurality of images into one image, but in the present embodiment, thereconstruction image is generated by averaging all intermediate images.

The output device 767 outputs the generated reconstruction image tooutside (the display 35, the memory 80, the removable media 38 or thelike), and stores the generated reconstruction image as the image forviewing.

The process performed by the digital camera 1 of the present embodimentwill be described with reference to a flowchart. When the digital camera1 is powered on, and the input device 50 receives an operation forpreparing a photographing, the digital camera 1 starts an image outputprocess as is the case in the digital camera 1 in the embodiment 1. Theimage output process in the present embodiment performs the image outputprocess 1 of FIG. 6 as is the case in the embodiment 1, except for usingthe image generation process 6 illustrated in FIG. 28 as the main imagegeneration process performed at step S109.

The image generation process 6 performed at step S109 will be describedwith reference to FIG. 28. Prior to the image generation process 6, thedigital camera 1 has imaged the photographic object and has stored(captured) the data of the light field image obtained by the imaging inthe RAM 23, the memory 34 or the like.

At step S1001, the seventh reconstruction image generator 760 retrievesfrom the ROM 22 the size of the partial image to be clipped from thesub-image, and sets (determines) the retrieved size as a clipping size.A correspondence table for the re-focal length and the clipping size hasbeen stored in the ROM 22. The clipping size is determined according tothe focal lengths of the main lens and the micro lenses in the microlens array, the position on which the imaging element 33 is arranged,and the re-focal length.

The difference of the angles of view for every micro lens is smaller asthe photographic object exists further away, and thus movement of theimaging location in the sub-images is also small. According to thisface, the clipping size is made small. On the contrary, the differenceof the angles of view for every micro lens is larger as the photographicobject exists nearer, and thus the movement of the imaging location inthe sub-image is also large. According to this face, the clipping sizeis made large. Specifically, it is assumed that the focal lengths of themain lens and the micro lenses are constant, and the correspondencetable of the reconstruction distance and the clipping size is set sothat the clipping size is smaller as the reconstruction distance islarger (refer to drawing 27B).

At step S1002, the clipping position setter 761 sets (determines) theposition of the partial image to be clipped from the sub-image. In thepresent embodiment, an initial value (setting position in a first loop)is the center of the sub-image.

Next, at S1003, the sub-image extractor 762 selects the sub-image to beprocessed. Specifically, k4 represents a counter variable, and the k4-thsub-image in the light field image LFI is extracted as a targetsub-image (step S1003).

Next, at step S1004, the partial area definer 763 and the partial imageextractor 764 clip the partial image from the selected sub-image.Specifically, the partial area definer 763 defines the area with thesize defined at step S1001, on a part corresponding to the clippingposition set at step S1002 on the target sub-image, as the partial area.Then, the partial image extractor 764 clips (extracts) the image of thepartial area (step S1004). Furthermore, at step S1004, the partial imageextractor 764 resizes the clipped image into the partial image with apredetermined number of pixels (for example, 4×4 pixels).

Next, at step S1005, the image arranger 765 arranges the partial imageclipped at step S1004 on the intermediate image, based on the clippingsize, the clipping position in the sub-image, and an alignment positionof the sub-image in the light field image. The concrete process isdescribed below using a concrete example. In this case, in the lightfield image photographed by the optical system, as illustrated in FIG.3, in which a focal point of the main lens is positioned in the mainlens side from the imaging element, each partial image is inverted infour directions to arrange on the intermediate image. In the case of theoptical system in which a focal point of the main lens is positionedbehind the imaging element, the partial images are not inverted in thefour directions.

At step S1006, the seventh reconstruction image generator 760 determineswhether all sub-images have been processed. When the determining resultis No, the seventh reconstruction image generator 760 increments thecounter variable k4 (step S1007), and returns to step S1003 to repeatthe process. When the determining result is Yes (step S1006; Yes), atstep S1008, the seventh reconstruction image generator 760 stores theintermediate image in the RAM 23 or the memory 34 (step S1008).

Next, at step S1008, the seventh reconstruction image generator 760determines whether a predetermined number (for example, eight pieces) ofintermediate images have been generated. When the determining result isNo (step S1009; No), the seventh reconstruction image generator 760returns to step S1002, and repeats the process with respect to the nextclipping position in which an offset is added to the last clippingposition. Although the offset can be set arbitrarily, it is desirable toset the offset value so that the partial area does not protrude from thesub-image, and so that the partial area can be set without bias from thesub-image according to the shape (a circular pattern in FIG. 29) of adiaphragm.

The seventh reconstruction image generator 760 repeats the process fromstep S1002 to step S1009. In this case, the seventh reconstruction imagegenerator 760 adds an offset of the clipping position, and arranges thepartial image on the intermediate image. It will be describespecifically below. In this way, the photographic object's positions arealigned between the intermediate images. When the determining result isYes at step S1009 (step S1009; Yes), at step S1010, the combiner 766averages the plurality of obtained intermediate images to generate thereconstruction image. After that, the digital camera 1 stores anddisplays the generated reconstruction image, and the image outputprocess 1 is terminated.

The process from step S1002 to step S1009 will be described using FIG.29 to FIG. 31. The process (step S1003 to step S1006) which generates afirst intermediate image is illustrated in FIG. 29. The sub-image S₁₁ atthe upper left of the light field image LFI is selected first (S1003).The partial image PI₁₁ is clipped (extracted) from the central part(initial value of the clipping position) of the selected sub-image S₁₁(S1004), and the partial image PI₁₁ is arranged at the upper left of theintermediate image ImI₁ (S1005). In FIG. 29 to FIG. 32, the referenceposition where the intermediate image ImI₁ is arranged is illustrated bythe dotted line and the dot.

In the following loop, the sub-image S₁₂ on the right is selected(S1003). The partial image PI₁₂ is clipped from the central part of theselected sub-image S₁₂ (S1004), and the partial image PI₁₂ is arrangedon the right of partial image PI₁₁ which has been arranged in theintermediate image ImI₁ (S1005). Such operations are repeated to theright end of the light field image LFI. When the process for thesub-image S_(IN) at the right end is completed, it returns to the leftend of the light field image LFI, and selects the sub-image S₂₁ on thesecond row from the top (step S1003). Then, the partial image PI₁₂ isclipped from the central part of the selected sub-image S₁₂ (S1004), andthe partial image PI₁₂ is arranged under the partial image PI₁₁ whichhas been arranged first in the intermediate image 1 (S1005). Suchoperations are repeated to the right end of the light field image LFI,as is the case for the first row. When the process for the sub-image Sat right end is completed, the sub-images S on the following row in thelight field image LFI will be processed. These operations are repeated.When it is determined that the sub-image S_(MN) at the lowermost andrightmost of light field image LFI is completed (S1006; Yes), theintermediate image ImI₁ is stored in the memory 34 or the like (S1008).Then, the generation of the intermediate image ImI₁ is terminated.

Since the partial images are arranged in order simply on theintermediate image ImI₁ in this phase, the connecting part of thesub-images is noticeable between each partial image.

FIG. 30 illustrates an example, in the second loop, of the process whichsets the clipping position on the upper left side in the sub-image, andgenerates the intermediate image ImI₂. In this case, the offset valuebetween the first loop and the second loop is a value to move theclipping position from the center of the sub-image to upper left side(for example, 10 pixels to the left and 10 pixels upward). In accordancewith a movement of clipping position to the upper left side rather thanthe case illustrated in FIG. 29, the position where the image arranger765 arranges the partial image on the intermediate image ImI₂ at stepS1005 is moved to the upper left side (FIG. 30 lower side).

The process which generates the intermediate image while adding theoffset to the clipping position (step S1002 to step S1009) are repeated.FIG. 31 illustrates an example of the process which sets the clippingposition at lower right in n-th loop, and generates the intermediateimage ImI_(n). From the second loop to the n-th loop, the offset valueis changed so that the clipping position moves gradually from the upperleft to the lower right (for example, 3 pixels to the right when notchanging the row, 3 pixels downward and 21 pixels on the left whenchanging the row, or the like). In accordance with a movement ofclipping position to the lower right side rather than the caseillustrated in FIG. 29, the position where the image arranger 765arranges the partial image on the intermediate image ImI_(n) at stepS1005 is moved to the lower right side (FIG. 31 lower side).

In the reconstruction image obtained by averaging a plurality ofintermediate images in this manner, the connecting part of thesub-images is not noticeable comparison with the intermediate image inwhich the partial images are simply arranged. In other words, at thetime of reconstructing an image by clipping an image from the sub-imageof the light field image, the clipping position is caused to change foreach intermediate image, and the intermediate images are averaged, andtherefore it is possible to obtain a comfortable (image with goodquality) and good reconstruction image.

In the present embodiment, the reconstruction image is obtained bydetermining the clipping size according to the reconstruction distance,changing the clipping position for each intermediate image, andaveraging the intermediate images. FIG. 32A to FIG. 32C illustratecomparisons of the intermediate images ImI₁, ImI₂ and ImI_(n) in FIG. 29to FIG. 31, where the clipping positions are different. When attentionis paid to the pixel with mark x, it can be understood that, in theintermediate images ImI₁, ImI₂ and ImI_(n), the pixels on differentpositions of the different sub-image are adopted, respectively. In otherwords, the present embodiment reconstructs the pixel of photographicobject to be reconstructed by averaging pixels on different positions ofdifferent sub-images according to a re-focal length. Moreover, since thepositions of the connecting part between the partial images aredifferent in ImI₁ to ImI_(n) respectively, combining the images(additional average) allows a generation of the reconstruction imagewith high image quality, in which sections between the partial imagesare not noticeable, with a small calculation amount for arrangement ofthe partial images and averaging.

In the present embodiment, alignment is performed taking the offset ofthe partial image into consideration when arranging the partial imageson each intermediate image, but alignment may be performed taking theoffset of the clipping position into consideration when averagingintermediate images. In other words, at the time of generating theintermediate images, the partial images are arranged on one imageirrespective of the clipping position. Then, at the time when thecombiner 766 combines the intermediate images, the intermediate imagesare shifted with each other according to the offset value and averaged.By this means, a similar effect can be provided.

The reconstruction distance may be set on the basis of manual operation,or may be set by measuring a distance to the photographic object whichis desired to focusing. Furthermore, the re-focal length may be setagain by reselecting a photographic object which is desired to focusinga touch panel or the like after displaying the reconstruction image.

In the present embodiment, although the repetition process whilechanging a clipping position is terminated by having created apredetermined number of intermediate images, the repetition process maybe terminated by a user's operation. For example, whenever the seventhreconstruction image generator 760 generates one intermediate image, thereconstruction image in which the intermediate images generated by thenare combined is displayed on the display to provide the image to theuser, and promoting user to operate a given handler (for example, ashutter button) when the user is satisfied with image quality. Then, therepetition process may be terminated by the operation of the handler. Inthis way, even if throughput of the CPU is low, the image quality withwhich the user is satisfied in the shortest time can be achieved.

Subsequently, a modification of the present embodiment will bedescribed. The present modification is an example which achieves thesimilar effect with the embodiments, even if the order of steps in theimage generation process 6 is changed. The main image generator (aneighth reconstruction image generator) of the present modification hassame functional elements in the seventh reconstruction image generator760 illustrated in FIG. 27A.

FIG. 33 is a flowchart of a process (an image generation process 7)performed by the main image generator (the eighth reconstruction imagegenerator) according to the present modification. Before performing theimage generation process 7 and before arriving at step S1101 of theimage generation process 7, the digital camera of the modificationperforms the process as is the case in step S1001 of FIG. 28.

At step S1102, the sub-image extractor of the eighth reconstructionimage generator selects the sub-image to be processed. Specifically, k5represents a counter variable, and k5-th sub-image in the light fieldimage LFI is extracted as a target sub-image (step S1102). At stepS1103, the clipping position setter sets (determines) a position of thepartial image to be clipped (extracted) from the sub-image as is thecase in step S1002 of the image generation process 6 (FIG. 28). Aninitial position is the center of the sub-image. At step S1104, thepartial area definer and the partial image extractor clip the partialimage from the sub-image selected as is the case in step S1004 of theimage generation process 6 (FIG. 28).

At step S1105, the image arranger adds the partial image to thereconstruction image based on the clipping size, the clipping positionin the sub-image, and the alignment position of the sub-image in thelight field image. Specific process is described below. At step S1106,the eighth reconstruction image generator determines whether apredetermined number (for example, 8) of clipping positions have beenprocessed. When a determination result is No (step S1106; No), the CPU11 returns to step S1103, changes the clipping position as is the casein the image generation process 6, and repeats the process. Suchrepetition results in setting of a plurality of clipping positions inthe overall process.

The boundary (circle in upper side of FIG. 34) of the sub-images dependson a shape of the diaphragm located in the optical system. In view ofthis matter, a range of changing the clipping position of the partialimage is restricted according to the shape of the diaphragm. Forexample, when the diaphragm has a substantially circular shape, and theclipped partial image has a quadrangle shape, at the clipping positionsetting process of step S1103, the offset value is set so that the rangeof changing the clipping position in a diagonal direction of the partialimage is smaller than a range of changing in a vertical and/orhorizontal direction of the sub-image.

When the determination result is Yes at step S1106 (step S1106; Yes),the eighth reconstruction image generator determines, at step S1107,whether all sub-images have been processed. When the determinationresult is No (step S1107; No), the eighth reconstruction image generatorincrements the counter variable k5 (step S1108), and after that, returnsto step S1102, selects following sub-image, and repeats the process.

When the determination result is Yes (step S1107; Yes), at step S1109,the combiner of the eighth reconstruction image generator adjusts theluminance of the reconstruction image. In other words, luminance isadjusted by dividing the pixel value which is increased by repetition ofthe addition by a setting number of the clipping position(normalization). After that, the digital camera 1 stores and displaysthe generated reconstruction image, and the image output process ofpresent modification is terminated.

The concept (outline) of the process of step S1103 to step 1106 of theimage generation process 7 will be described using FIG. 34 and FIG. 35.

The description is made using an example of process for the sub-imageS11 at the upper left on the light field image LFI. At step S1103 in afirst loop, the clipping position (central part) in the selectedsub-image S11 is set. The partial image PI₁₁ is clipped (extracted) fromthe selected sub-image S₁₁ (S1104). The clipped partial image PI₁₁ isadded to the reconstruction image RI based on the clipping size, theclipping position in the sub-image S₁₁, and the alignment position ofthe sub-image S₁₁ in light field image LFI (S1105). The square in lowerside of FIG. 34 illustrates this addition position.

In the following loop (FIG. 35), a new clipping position (upper leftfrom the central part) in which the offset is added is set (S1103). Inthis loop, the clipping size and the alignment position of the sub-imageS₁₁ in the light field image LFI are the same as the size and positionin FIG. 34, but the clipping positions in the sub-image S₁₁ aredifferent.

The light field image LFI is a series of images including a plurality ofsub-images S₁₁ to S_(NM). Therefore, depending on a inapplicable settingof the offset, not only an area which is effective as the sub-image Sbut also an area which is invalid as the sub-image may be clipped as thepartial images. The partial image including the invalid area serves as anoise source, and therefore the clipping range (the partial area) isrequired to be set within a range formed by the boundary of thesub-image S. Therefore, it is necessary to restrict the offset amount ofthe clipping position of the partial image according to the shape of thediaphragm. For example, when the diaphragm has a substantially circularshape, and the clipped partial image has a quadrangle shape, the rangeof changing the clipping position in the diagonal direction of thepartial image is made smaller than the range of changing in the verticaland/or horizontal direction of the sub-image. As a concrete example,when the clipping position is changed from the central part which is setas the starting point, a possible offset value in the vertical and/orhorizontal direction may be set larger, and a possible offset value inthe diagonal direction may be set smaller.

The position where the partial image is added to the reconstructionimage RI is also applied the offset with the offset amount of theclipping position. FIG. 35 illustrates a first addition position with adotted line, and a second addition position with a solid line, for thesake of illustrating the addition positions which are applied theoffset. On the reconstruction image RI, the partial images are added sothat the boundary of the partial images do not overlap, and thus theconnecting part of a plurality of sub-images is made not noticeable anda good (high image quality) reconstruction image RI can be obtained.

Although the embodiments 1 to 5 of the present invention have beendescribed, but the embodiments of the present invention is not limitedto the above-described embodiments, and various modifications can bemade.

Although the above-described embodiments are described taking an examplethat the images are gray scale images, the images used as the processingobject in the present invention are not limited to the gray scale image.For example, the images may be RGB images in which three pixel values ofR (red), G (green) and B (blue) are defined for each pixel. In thiscase, pixel values are similarly processed with a vector of RGB.Alternatively, the above-described process may be applied to each ofvalues R, G, and B respectively, by handling each value of RGB asindependent gray scale image. According to this configuration, thereconstruction image which is a color image can be generated from thelight field image which is a color image.

The reconstruction distance (which corresponds to a distance to thereconstruction surface) may be set on the basis of manual operation, ormay be set by measuring a distance to the photographic object which isdesired to focusing. Furthermore, the re-focal length may be set againby reselecting a photographic object which is desired to focusing atouch panel or the like after displaying the reconstruction image.

Moreover, although a case where the main lens imaging plane MA exists inthe main lens side from the micro lens array is described, theconfiguration of the optical system of the present invention is notlimited to such configuration.

For example, the main lens imaging plane MA may exist on the back of themicro lens array, and the others are same design. In this case, whenarranging the image for arrangement on the generation image, the imageis arranged as it is, without inverting in four directions.

Moreover, a case may be presumable in which the imaging plane of themicro lens is on the micro lens side from the imaging element. In thiscase, the arrangement image is further inverted in four directions, andis arranged on the generation image, on the basis of a case other thanthis case.

The live view image generation process, the confirmation imagegeneration process, and the main image generation process are notlimited to above-described combination. In the present invention, acombination of the above-described processes can be arbitrarily selectedunder the condition that a required calculation amount for the live viewimage generation process is equal to or less than a calculation amountfor the confirmation image generation process, and a requiredcalculation amount for the main image generation process is larger thana calculation amount for the confirmation image generation process.

For example, the live view image generation process and the confirmationimage generation process may be the above-described image generationprocess 1, and the main image generation process may be the imagegeneration process 3 or 4. Alternatively, a combination that, the liveview image generation process and the confirmation image generationprocess are the image generation processes 2, and the main imagegeneration process is the image generation process 3 or 4, may beemployed.

Moreover, the same type of image generation process may be adopted bythe live view image generation process and the confirmation imagegeneration process, and the calculation amounts may be adjusted in theprocesses. For example, when the image generation process 2 is adoptedby the live view image generation process and the confirmation imagegeneration process, the live view image generation process calculatesthe image displacement degrees only in one direction, and theconfirmation image generation process calculates the image displacementdegrees in two directions of vertical and horizontal directions, andthereby allowing a configuration which improves the accuracy of theimage displacement degrees with respect to the confirmation image.

Alternatively, the confirmation image generation process may take thelarger area for calculating the image displacement degree than the areain the live view generation process, and thereby allowing aconfiguration which improves the accuracy of the image displacementdegrees.

According to such configuration, the live view image which is requiredto be displayed most rapidly can be generated at high speed using leastcalculations according to a speed of the CPU. The confirmation image canbe displayed sufficiently with the image quality at least equivalent tothe live view image, or with higher image quality depending on setting.Furthermore, as for the main image, it is possible to obtain a highquality image. Thus, a digital camera with higher convenience for a usercan be provided by using different image generation processes accordingto a required display speed.

In addition, the hardware configurations and flowcharts are examples,and modification and correction can be made arbitrarily.

For example, in the present embodiments and the modifications of theembodiments, the image generation process is performed by thereconstruction image generator included in the digital camera 1.However, the image generation process (particularly a process performedby the main image generator) may be performed in an image processingdevice (for example, PC which has installed a program performing theimage generation process) other than the digital camera 1. In this case,the light field image generated in another imaging device may beacquired and read by the image processing device, and this imageprocessing device may perform the reconstruction process (the imagegeneration process 6 or 7). In this case, another imaging device maystore the position of the photographic object selected by the touchpanel and the light field image in the removable media or the like.Accordingly a reconstruction in the image processing device isfacilitated. Moreover, the photographic object distance measured at thetime of photographing and the light field image are stored in advance inthe removable media or the like so that these data can be utilized atthe time of reconstruction.

The core elements for performing the process of the image generation,which includes the CPU 21, the ROM 22, the RAM 23 and the like, can beimplemented using a general computer system, other than a dedicatedsystem. For example, a computer program for performing theabove-described operation is distributed by a computer readablerecording medium (a flexible disk, CD-ROM, DVD-ROM, or the like) inwhich the program is stored, and the computer program is installed in acomputer to implement a part for performing the above-describeddescribed image generation process. Alternatively, the computer programis stored in advance in a storage device included in a server located ona communication network such as the Internet, and a general computersystem downloads the program to implement the part for performing theabove-described described image generation process.

In cases where OS (operating system) and an application program sharesand realizes the functions for performing the above-described imagegeneration process or in cases where the OS and the application programrealizes the functions in cooperation, only the application program maybe stored in a recording medium or a storage device.

Moreover, it is also possible to distribute the computer program througha communication network. For example, the computer program may be placeson the bulletin board system (BBS) on the communication network, and maybe distributed through the network. The system may be constituted sothat the above-described processes can be executed, by activating thiscomputer program and executing the program as well as other applicationprograms under a control of OS.

Although the embodiments of the present invention are described, theembodiments are only exemplification and does not limit the technicalscope of the present invention. The present invention can employ othervarious embodiments, and various modifications, such as an omission andsubstitution, can be made within a range not departing from the gist ofthe present invention. These embodiments and the modifications thereofare included in the scope and gist of the invention described in thespecification and the like, and in the invention specified in the claimsand equivalent thereof.

[Industrial Availability]

The present invention can apply to an image processing device which canphotograph a multi-view image and an image processing device whichprocesses a multi-view image.

Having described and illustrated the principles of this application byreference to one or more preferred embodiments, it should be apparentthat the preferred embodiments may be modified in arrangement and detailwithout departing from the principles disclosed herein and that it isintended that the application be construed as including all suchmodifications and variations insofar as they come within the spirit andscope of the subject matter disclosed herein.

What is claimed is:
 1. An image processing device comprising: an image obtainer that obtains a multi-view image in which sub-images from viewpoints are aligned; a clipper that clips partial images from the sub-images; a generator that generates a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; a distance setter that sets a re-focal length of the reconstruction image; an interval determiner that determines an interval, at which the partial images are arranged on the reconstruction image, according to the re-focal length; and a size determiner that determines a clipping size of the partial images so that the partial images are overlapped on the reconstruction image.
 2. The image processing device according to claim 1, further comprising: a blurring degree determiner that determines a blurring degree, wherein the size determiner determines a larger clipping size as the blurring degree is larger.
 3. An image processing device comprising: an image obtainer that obtains a multi-view image in which sub-images from viewpoints are aligned; a clipper that clips partial images from the sub-images; a generator that generates a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; a size determiner that determines a clipping size of the partial images; and a position determiner that determines clipping positions of the partial images, wherein the generator combines partial images corresponding to the clipping positions based on the clipping size and the alignment position of the sub-images, to generate the reconstruction image.
 4. The image processing device according to claim 3, further comprising: a distance setter that sets a re-focal length of the reconstruction image, wherein the size determiner determines the clipping size according to the re-focal length.
 5. The image processing device according to claim 4, wherein the size determiner determines a smaller clipping size as the re-focal length is larger.
 6. The image processing device according to claim 3, wherein the position determiner determines the clipping positions according to a shape of a diaphragm by which the multi-view image is photographed.
 7. The image processing device according to claim 3, wherein: the generator includes: an intermediate image generator that generates an intermediate image each time the position determiner determines the clipping positions, by arranging the partial images on the intermediate image based on the alignment position of the sub-images and the clipping positions currently determined by the position determiner; and a combiner that combines each of the generated intermediate images to generate the reconstruction image; and the image processing device further comprises a display that displays the reconstruction image in which the intermediate images are combined each time the position determiner determines the clipping positions.
 8. An image processing device comprising: an image obtainer that obtains a multi-view image in which sub-images from viewpoints are aligned; a clipper that clips partial images from the sub-images; a generator that generates a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; an image displacement obtainer that obtains image displacement degrees for the sub-images respectively, wherein each of the image displacement degrees indicates a displacement between a position of a predetermined part in the sub-image and a position of a part corresponding to a photographic object captured in the predetermined part in another sub-image; and a size determiner that determines a larger clipping size for the partial images with a larger image displacement degree.
 9. The image processing device according to claim 8, wherein the image displacement obtainer calculates an image displacement degree for a sub-image selected as a calculation object from among the sub-images, and estimates the image displacement degree for a sub-image other than the calculation object based on the calculated image displacement degree.
 10. The image processing device according to claim 9, wherein the image obtainer sequentially obtains multi-view images, the image displacement obtainer selects the sub-images for the calculation object in a cyclic manner, for each of the multi-view images sequentially obtained, to obtain the image displacement degrees for the sub-images, and the generator sequentially generates the reconstruction image from the sequentially obtained multi-view images using the sequentially obtained image displacement degrees.
 11. An image processing device comprising: an image obtainer that obtains a multi-view image in which sub-images from viewpoints are aligned; a clipper that clips partial images from the sub-images; a generator that generates a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; and a size determiner that determines a clipping size of the partial images to fit arrangement areas which are obtained by dividing the reconstruction image according to the sub-images.
 12. An image processing device comprising: an image obtainer that obtains a multi-view image in which sub-images from viewpoints are aligned; a clipper that clips partial images from the sub-images; a generator that generates a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; and a size determiner that determines a clipping size of the partial images, wherein the generator (a) divides the reconstruction image into arrangement areas corresponding to the sub-images, (b) reduces or expands the partial images to fit a size of a corresponding arrangement area, and (c) arranges the partial images.
 13. An image processing device comprising: an image obtainer that obtains a multi-view image in which sub-images from viewpoints are aligned; a first selector that selects a first image from the sub-images; a pixel selector that selects a first pixel included in the first image; a second selector that selects a second image from the sub-images; a searcher that searches a second pixel corresponding to the first pixel among pixels included in the second image; a pixel processor that processes the sub-images by applying a blurring addition process to the first pixel and the second pixel in accordance with a blurring intensity, the blurring intensity being based on a positional relationship between the first pixel and the second pixel and a re-focal length; and a reconstructor that reconstructs an image from the sub-images processed by the pixel processor.
 14. The image processing device according to claim 13, wherein the reconstructor includes: a clipper that clips partial images from the sub-images; and a generator that arranges the partial images based on an alignment position of the sub-images corresponding to the partial images to generate the reconstruction image.
 15. An image generating method comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; clipping partial images from the sub-images; setting a re-focal length of a reconstruction image; and generating the reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; wherein generating the reconstruction image comprises determining an interval, at which the partial images are arranged on the reconstruction image, according to the re-focal length; and wherein clipping the partial images comprises determining a clipping size of the partial images so that the partial images are overlapped on the reconstruction image.
 16. A non-transitory computer-readable storage medium having a computer-executable program stored thereon, the program being executable to control a computer to execute functions comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; clipping partial images from the sub-images; setting a re-focal length of a reconstruction image; and generating the reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; wherein generating the reconstruction image comprises determining an interval, at which the partial images are arranged on the reconstruction image, according to the re-focal length; and wherein clipping the partial images comprises determining a clipping size of the partial images so that the partial images are overlapped on the reconstruction image.
 17. An image generating method comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; clipping partial images from the sub-images; and generating a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; wherein clipping the partial images comprises determining a clipping size of the partial images and clipping positions of the partial images; and wherein generating the reconstruction image comprises combining partial images corresponding to the clipping positions based on the clipping size and the alignment position of the sub-images, to generate the reconstruction image.
 18. An image generating method comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; obtaining image displacement degrees for the sub-images respectively, wherein each of the image displacement degrees indicates a displacement between a position of a predetermined part in the sub-image and a position of a part corresponding to a photographic object captured in the predetermined part in another sub-image; clipping partial images from the sub-images, wherein a larger clipping size is determined for the partial images with a larger image displacement degree; and generating a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images.
 19. An image generating method comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; clipping partial images from the sub-images; and generating a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; wherein clipping the partial images comprises determining a clipping size of the partial images to fit arrangement areas which are obtained by dividing the reconstruction image according to the sub-images.
 20. An image generating method comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; clipping partial images from the sub-images; and generating a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; wherein clipping the partial images comprises determines a clipping size of the partial images; and wherein generating the reconstruction image comprises (a) dividing the reconstruction image into arrangement areas corresponding to the sub-images, (b) reducing or expanding the partial images to fit a size of a corresponding arrangement area, and (c) arranging the partial images.
 21. An image generating method comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; selecting a first image from the sub-images; selecting a first pixel included in the first image; selecting a second image from the sub-images; searching a second pixel corresponding to the first pixel among pixels included in the second image; processing the sub-images by applying a blurring addition process to the first pixel and the second pixel in accordance with a blurring intensity, the blurring intensity being based on a positional relationship between the first pixel and the second pixel and a re-focal length; and reconstructing an image from the sub-images having been processed.
 22. A non-transitory computer-readable storage medium having a computer-executable program stored thereon, the program being executable to control a computer to execute functions comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; clipping partial images from the sub-images; and generating a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; wherein clipping the partial images comprises determining a clipping size of the partial images and clipping positions of the partial images; and wherein generating the reconstruction image comprises combining partial images corresponding to the clipping positions based on the clipping size and the alignment position of the sub-images, to generate the reconstruction image.
 23. A non-transitory computer-readable storage medium having a computer-executable program stored thereon, the program being executable to control a computer to execute functions comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; obtaining image displacement degrees for the sub-images respectively, wherein each of the image displacement degrees indicates a displacement between a position of a predetermined part in the sub-image and a position of a part corresponding to a photographic object captured in the predetermined part in another sub-image; clipping partial images from the sub-images, wherein a larger clipping size is determined for the partial images with a larger image displacement degree; and generating a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images.
 24. A non-transitory computer-readable storage medium having a computer-executable program stored thereon, the program being executable to control a computer to execute functions comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; clipping partial images from the sub-images; and generating a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; wherein clipping the partial images comprises determining a clipping size of the partial images to fit arrangement areas which are obtained by dividing the reconstruction image according to the sub-images.
 25. A non-transitory computer-readable storage medium having a computer-executable program stored thereon, the program being executable to control a computer to execute functions comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; clipping partial images from the sub-images; and generating a reconstruction image by arranging the partial images based on an alignment position of the sub-images corresponding to the partial images; wherein clipping the partial images comprises determines a clipping size of the partial images; and wherein generating the reconstruction image comprises (a) dividing the reconstruction image into arrangement areas corresponding to the sub-images, (b) reducing or expanding the partial images to fit a size of a corresponding arrangement area, and (c) arranging the partial images.
 26. A non-transitory computer-readable storage medium having a computer-executable program stored thereon, the program being executable to control a computer to execute functions comprising: obtaining a multi-view image in which sub-images from viewpoints are aligned; selecting a first image from the sub-images; selecting a first pixel included in the first image; selecting a second image from the sub-images; searching a second pixel corresponding to the first pixel among pixels included in the second image; processing the sub-images by applying a blurring addition process to the first pixel and the second pixel in accordance with a blurring intensity, the blurring intensity being based on a positional relationship between the first pixel and the second pixel and a re-focal length; and reconstructing an image from the sub-images having been processed. 